[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929496330398-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:24:56 dist-test-slave-nk6d krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:24:56 dist-test-slave-nk6d krb5kdc[399](info): set up 2 sockets
Jun 20 04:24:56 dist-test-slave-nk6d krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260620 04:24:57.296205 392 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929496330398-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929496330398-392-0/krb5kdc/test-user.keytab.
Jun 20 04:24:57 dist-test-slave-nk6d krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929497, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:24:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:24:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 04:24:57.406797 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929496330398-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929496330398-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929496330398-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39111
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39111
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:24:57.512900 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:24:57.513291 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:24:57.513379 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:24:57.517407 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:24:57.517463 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:24:57.517490 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:24:57.517513 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:24:57.517539 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:24:57.522714 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929496330398-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929496330398-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39111
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929496330398-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:24:57.523746 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:24:57.524783 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:24:57.533761 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:24:57.533814 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:24:57.533864 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:24:57.534003 412 server_base.cc:1034] running on GCE node
I20260620 04:24:57.534606 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:24:57.535677 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:24:57.536830 412 hybrid_clock.cc:648] HybridClock initialized: now 1781929497536764 us; error 79 us; skew 500 ppm
Jun 20 04:24:57 dist-test-slave-nk6d krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929497, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:24:57.542788 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 04:24:57.543913 412 webserver.cc:460] Webserver started at http://127.0.98.62:34989/ using document root <none> and password file <none>
I20260620 04:24:57.544600 412 fs_manager.cc:359] Metadata directory not provided
I20260620 04:24:57.544755 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:24:57.545012 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:24:57.547732 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929496330398-392-0/minicluster-data/master-0/data/instance:
uuid: "666035c47e3844fab4b7c5e3fbb04662"
format_stamp: "Formatted at 2026-06-20 04:24:57 on dist-test-slave-nk6d"
server_key: "7848065f6ca3f6afedeccc459a8252f9"
server_key_iv: "830ea5f6b7aa29dfecb905a6036526da"
server_key_version: "encryptionkey@0"
I20260620 04:24:57.548415 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929496330398-392-0/minicluster-data/master-0/wal/instance:
uuid: "666035c47e3844fab4b7c5e3fbb04662"
format_stamp: "Formatted at 2026-06-20 04:24:57 on dist-test-slave-nk6d"
server_key: "7848065f6ca3f6afedeccc459a8252f9"
server_key_iv: "830ea5f6b7aa29dfecb905a6036526da"
server_key_version: "encryptionkey@0"
I20260620 04:24:57.552783 412 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 04:24:57.555240 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:24:57.556587 412 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 04:24:57.556701 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929496330398-392-0/minicluster-data/master-0/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929496330398-392-0/minicluster-data/master-0/wal
uuid: "666035c47e3844fab4b7c5e3fbb04662"
format_stamp: "Formatted at 2026-06-20 04:24:57 on dist-test-slave-nk6d"
server_key: "7848065f6ca3f6afedeccc459a8252f9"
server_key_iv: "830ea5f6b7aa29dfecb905a6036526da"
server_key_version: "encryptionkey@0"
I20260620 04:24:57.556800 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929496330398-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929496330398-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929496330398-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:24:57.566128 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:24:57.568924 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:24:57.569208 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:24:57.575760 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39111
I20260620 04:24:57.575819 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39111 every 8 connection(s)
I20260620 04:24:57.576713 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:24:57.579378 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:24:57.582701 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 412
I20260620 04:24:57.582871 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929496330398-392-0/minicluster-data/master-0/wal/instance
I20260620 04:24:57.584491 392 external_mini_cluster.cc:1428] Setting key 52622c754689dc85c7c6e66fb0a878d3
I20260620 04:24:57.585449 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 666035c47e3844fab4b7c5e3fbb04662: Bootstrap starting.
I20260620 04:24:57.588256 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 666035c47e3844fab4b7c5e3fbb04662: Neither blocks nor log segments found. Creating new log.
I20260620 04:24:57.588961 480 log.cc:824] T 00000000000000000000000000000000 P 666035c47e3844fab4b7c5e3fbb04662: Log is configured to *not* fsync() on all Append() calls
I20260620 04:24:57.590641 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 666035c47e3844fab4b7c5e3fbb04662: No bootstrap required, opened a new log
I20260620 04:24:57.594722 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P 666035c47e3844fab4b7c5e3fbb04662 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "666035c47e3844fab4b7c5e3fbb04662" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39111 } }
I20260620 04:24:57.594944 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P 666035c47e3844fab4b7c5e3fbb04662 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:24:57.595000 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P 666035c47e3844fab4b7c5e3fbb04662 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 666035c47e3844fab4b7c5e3fbb04662, State: Initialized, Role: FOLLOWER
I20260620 04:24:57.595417 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P 666035c47e3844fab4b7c5e3fbb04662 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "666035c47e3844fab4b7c5e3fbb04662" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39111 } }
I20260620 04:24:57.595541 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P 666035c47e3844fab4b7c5e3fbb04662 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:24:57.595616 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P 666035c47e3844fab4b7c5e3fbb04662 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:24:57.595713 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 666035c47e3844fab4b7c5e3fbb04662 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:24:57.597015 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P 666035c47e3844fab4b7c5e3fbb04662 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "666035c47e3844fab4b7c5e3fbb04662" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39111 } }
I20260620 04:24:57.597364 480 leader_election.cc:302] T 00000000000000000000000000000000 P 666035c47e3844fab4b7c5e3fbb04662 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 666035c47e3844fab4b7c5e3fbb04662; no voters:
I20260620 04:24:57.597647 480 leader_election.cc:288] T 00000000000000000000000000000000 P 666035c47e3844fab4b7c5e3fbb04662 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:24:57.597743 485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 666035c47e3844fab4b7c5e3fbb04662 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:24:57.597961 485 raft_consensus.cc:686] T 00000000000000000000000000000000 P 666035c47e3844fab4b7c5e3fbb04662 [term 1 LEADER]: Becoming Leader. State: Replica: 666035c47e3844fab4b7c5e3fbb04662, State: Running, Role: LEADER
I20260620 04:24:57.598261 485 consensus_queue.cc:237] T 00000000000000000000000000000000 P 666035c47e3844fab4b7c5e3fbb04662 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "666035c47e3844fab4b7c5e3fbb04662" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39111 } }
I20260620 04:24:57.598758 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P 666035c47e3844fab4b7c5e3fbb04662 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:24:57.599862 486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 666035c47e3844fab4b7c5e3fbb04662 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "666035c47e3844fab4b7c5e3fbb04662" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "666035c47e3844fab4b7c5e3fbb04662" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39111 } } }
I20260620 04:24:57.600046 486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 666035c47e3844fab4b7c5e3fbb04662 [sys.catalog]: This master's current role is: LEADER
I20260620 04:24:57.600284 486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 666035c47e3844fab4b7c5e3fbb04662 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 666035c47e3844fab4b7c5e3fbb04662. Latest consensus state: current_term: 1 leader_uuid: "666035c47e3844fab4b7c5e3fbb04662" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "666035c47e3844fab4b7c5e3fbb04662" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39111 } } }
I20260620 04:24:57.600363 486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 666035c47e3844fab4b7c5e3fbb04662 [sys.catalog]: This master's current role is: LEADER
I20260620 04:24:57.600704 496 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Jun 20 04:24:57 dist-test-slave-nk6d krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929497, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:24:57.603938 496 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:24:57.609351 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:24:57.587818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38048 (local address 127.0.98.62:39111)
0620 04:24:57.588328 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0620 04:24:57.588347 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 04:24:57.589546 (+ 1199us) server_negotiation.cc:407] Connection header received
0620 04:24:57.592808 (+ 3262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:24:57.592832 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:24:57.593148 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:24:57.593479 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:24:57.595672 (+ 2193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:24:57.597236 (+ 1564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:24:57.598915 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:24:57.599581 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:24:57.599712 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:24:57.604063 (+ 4351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:24:57.604068 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:24:57.604080 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:24:57.604107 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:24:57.605647 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:24:57.606246 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:24:57.606252 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:24:57.606258 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:24:57.606396 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:24:57.606770 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:24:57.606774 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:24:57.606776 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:24:57.607279 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:24:57.607486 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0620 04:24:57.608076 (+ 590us) server_negotiation.cc:299] Negotiation successful
0620 04:24:57.608364 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":170,"threads_started":1}
I20260620 04:24:57.612907 496 catalog_manager.cc:1347] Generated new cluster ID: d5dbd6c91a93415890a1e8a675dd9e2c
I20260620 04:24:57.612967 496 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:24:57.634718 496 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:24:57.635643 496 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:24:57.640129 496 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 666035c47e3844fab4b7c5e3fbb04662: Generated new TSK 0
I20260620 04:24:57.640961 496 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929496330398-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929496330398-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 04:24:57.672036 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929496330398-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929496330398-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929496330398-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929496330398-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39111
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929496330398-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:24:57.779563 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:24:57.779847 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:24:57.779913 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:24:57.783900 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:24:57.783986 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:24:57.784062 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 04:24:57.789949 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929496330398-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929496330398-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929496330398-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929496330398-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39111
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929496330398-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:24:57.791011 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:24:57.792044 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:24:57.801489 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:24:57.801493 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:24:57.801779 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:24:57.801947 506 server_base.cc:1034] running on GCE node
I20260620 04:24:57.802297 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:24:57.802907 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:24:57.804052 506 hybrid_clock.cc:648] HybridClock initialized: now 1781929497804030 us; error 31 us; skew 500 ppm
Jun 20 04:24:57 dist-test-slave-nk6d krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929497, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:24:57.806689 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 04:24:57.807582 506 webserver.cc:460] Webserver started at http://127.0.98.1:37053/ using document root <none> and password file <none>
I20260620 04:24:57.808161 506 fs_manager.cc:359] Metadata directory not provided
I20260620 04:24:57.808225 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:24:57.808410 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:24:57.810395 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929496330398-392-0/minicluster-data/ts-0/data/instance:
uuid: "3ac26ffac06947a39319aa8ab66d69b9"
format_stamp: "Formatted at 2026-06-20 04:24:57 on dist-test-slave-nk6d"
server_key: "87b3b18263f1799a45b7d2c2520d1025"
server_key_iv: "a6d71dc53ad3479752480720797c7329"
server_key_version: "encryptionkey@0"
I20260620 04:24:57.810925 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929496330398-392-0/minicluster-data/ts-0/wal/instance:
uuid: "3ac26ffac06947a39319aa8ab66d69b9"
format_stamp: "Formatted at 2026-06-20 04:24:57 on dist-test-slave-nk6d"
server_key: "87b3b18263f1799a45b7d2c2520d1025"
server_key_iv: "a6d71dc53ad3479752480720797c7329"
server_key_version: "encryptionkey@0"
I20260620 04:24:57.814515 506 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260620 04:24:57.816828 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:24:57.818034 506 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:24:57.818135 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929496330398-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929496330398-392-0/minicluster-data/ts-0/wal
uuid: "3ac26ffac06947a39319aa8ab66d69b9"
format_stamp: "Formatted at 2026-06-20 04:24:57 on dist-test-slave-nk6d"
server_key: "87b3b18263f1799a45b7d2c2520d1025"
server_key_iv: "a6d71dc53ad3479752480720797c7329"
server_key_version: "encryptionkey@0"
I20260620 04:24:57.818221 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929496330398-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929496330398-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929496330398-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:24:57.840122 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:24:57.842844 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:24:57.843034 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:24:57.843593 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:24:57.844417 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:24:57.844463 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:24:57.844514 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:24:57.844542 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:24:57.852555 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40273
I20260620 04:24:57.852665 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40273 every 8 connection(s)
I20260620 04:24:57.853628 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929496330398-392-0/minicluster-data/ts-0/data/info.pb
I20260620 04:24:57.858144 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 506
I20260620 04:24:57.858232 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929496330398-392-0/minicluster-data/ts-0/wal/instance
I20260620 04:24:57.858453 392 external_mini_cluster.cc:1428] Setting key ad999ba849db53b06f9df8e878273a0f
Jun 20 04:24:57 dist-test-slave-nk6d krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929497, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 04:24:57.867480 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:24:57.855365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52497 (local address 127.0.98.62:39111)
0620 04:24:57.855500 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 04:24:57.855503 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:24:57.856357 (+ 854us) server_negotiation.cc:407] Connection header received
0620 04:24:57.857641 (+ 1284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:24:57.857644 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:24:57.857700 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:24:57.857776 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:24:57.859101 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:24:57.859712 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:24:57.860970 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:24:57.861242 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:24:57.861331 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:24:57.864014 (+ 2683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:24:57.864018 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:24:57.864020 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:24:57.864043 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:24:57.865488 (+ 1445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:24:57.865996 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:24:57.865999 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:24:57.866000 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:24:57.866072 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:24:57.866379 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:24:57.866382 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:24:57.866384 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:24:57.866570 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:24:57.866651 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 04:24:57.867181 (+ 530us) server_negotiation.cc:299] Negotiation successful
0620 04:24:57.867318 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 04:24:57.868407 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:24:57.855747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39111 (local address 127.0.98.1:52497)
0620 04:24:57.856210 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0620 04:24:57.856253 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 04:24:57.857433 (+ 1180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:24:57.857990 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:24:57.857997 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:24:57.858390 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:24:57.858961 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:24:57.858982 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:24:57.859823 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:24:57.859826 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:24:57.860864 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:24:57.860874 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:24:57.861374 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:24:57.861378 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:24:57.861556 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:24:57.862154 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:24:57.862170 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:24:57.863830 (+ 1660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:24:57.865598 (+ 1768us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:24:57.865604 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:24:57.865622 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:24:57.865901 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:24:57.866154 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:24:57.866157 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:24:57.866160 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:24:57.866295 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:24:57.866665 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:24:57.866672 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:24:57.866957 (+ 285us) client_negotiation.cc:769] Sending connection context
0620 04:24:57.867124 (+ 167us) client_negotiation.cc:240] Negotiation successful
0620 04:24:57.867488 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":134,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 04:24:57.869657 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39111
I20260620 04:24:57.869904 635 heartbeater.cc:464] Registering TS with master...
I20260620 04:24:57.870369 635 heartbeater.cc:511] Master 127.0.98.62:39111 requested a full tablet report, sending...
I20260620 04:24:57.871817 444 ts_manager.cc:194] Registered new tserver with Master: 3ac26ffac06947a39319aa8ab66d69b9 (127.0.98.1:40273)
I20260620 04:24:57.872916 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52497
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929496330398-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929496330398-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 04:24:57.882779 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929496330398-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929496330398-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929496330398-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929496330398-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39111
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929496330398-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:24:57.990561 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:24:57.990825 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:24:57.990868 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:24:57.994881 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:24:57.994956 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:24:57.995018 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 04:24:58.000463 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929496330398-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929496330398-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929496330398-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929496330398-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39111
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929496330398-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:24:58.001549 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:24:58.002511 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:24:58.011626 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:24:58.011626 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:24:58.011626 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:24:58.011940 640 server_base.cc:1034] running on GCE node
I20260620 04:24:58.012348 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:24:58.012953 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:24:58.014104 640 hybrid_clock.cc:648] HybridClock initialized: now 1781929498014081 us; error 35 us; skew 500 ppm
Jun 20 04:24:58 dist-test-slave-nk6d krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929498, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:24:58.016690 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 04:24:58.017544 640 webserver.cc:460] Webserver started at http://127.0.98.2:46771/ using document root <none> and password file <none>
I20260620 04:24:58.018076 640 fs_manager.cc:359] Metadata directory not provided
I20260620 04:24:58.018141 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:24:58.018395 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:24:58.020470 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929496330398-392-0/minicluster-data/ts-1/data/instance:
uuid: "3c6aad3108aa4199bbaad1f460879176"
format_stamp: "Formatted at 2026-06-20 04:24:58 on dist-test-slave-nk6d"
server_key: "46412aae5d5d0c03722b442d15cbec61"
server_key_iv: "64cf1260e41e42840327bf6517c2e229"
server_key_version: "encryptionkey@0"
I20260620 04:24:58.021056 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929496330398-392-0/minicluster-data/ts-1/wal/instance:
uuid: "3c6aad3108aa4199bbaad1f460879176"
format_stamp: "Formatted at 2026-06-20 04:24:58 on dist-test-slave-nk6d"
server_key: "46412aae5d5d0c03722b442d15cbec61"
server_key_iv: "64cf1260e41e42840327bf6517c2e229"
server_key_version: "encryptionkey@0"
I20260620 04:24:58.024927 640 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 04:24:58.027266 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:24:58.028620 640 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:24:58.028724 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929496330398-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929496330398-392-0/minicluster-data/ts-1/wal
uuid: "3c6aad3108aa4199bbaad1f460879176"
format_stamp: "Formatted at 2026-06-20 04:24:58 on dist-test-slave-nk6d"
server_key: "46412aae5d5d0c03722b442d15cbec61"
server_key_iv: "64cf1260e41e42840327bf6517c2e229"
server_key_version: "encryptionkey@0"
I20260620 04:24:58.028820 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929496330398-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929496330398-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929496330398-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:24:58.039853 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:24:58.042347 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:24:58.042503 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:24:58.043023 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:24:58.043761 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:24:58.043804 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:24:58.043853 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:24:58.043880 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:24:58.053062 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40407
I20260620 04:24:58.053082 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40407 every 8 connection(s)
I20260620 04:24:58.053939 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929496330398-392-0/minicluster-data/ts-1/data/info.pb
I20260620 04:24:58.058482 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 640
I20260620 04:24:58.058578 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929496330398-392-0/minicluster-data/ts-1/wal/instance
I20260620 04:24:58.058809 392 external_mini_cluster.cc:1428] Setting key 6c6b00847777262958016e073fe1c64b
Jun 20 04:24:58 dist-test-slave-nk6d krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929498, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:24:58.067682 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:24:58.055691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34563 (local address 127.0.98.62:39111)
0620 04:24:58.055833 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 04:24:58.055837 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:24:58.056752 (+ 915us) server_negotiation.cc:407] Connection header received
0620 04:24:58.058101 (+ 1349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:24:58.058104 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:24:58.058161 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:24:58.058230 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:24:58.059513 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:24:58.060115 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:24:58.061256 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:24:58.061471 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:24:58.061555 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:24:58.064117 (+ 2562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:24:58.064122 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:24:58.064124 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:24:58.064153 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:24:58.065695 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:24:58.066201 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:24:58.066204 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:24:58.066206 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:24:58.066291 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:24:58.066588 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:24:58.066591 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:24:58.066593 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:24:58.066813 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:24:58.066909 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 04:24:58.067375 (+ 466us) server_negotiation.cc:299] Negotiation successful
0620 04:24:58.067512 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 04:24:58.068506 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:24:58.056107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39111 (local address 127.0.98.2:34563)
0620 04:24:58.056596 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0620 04:24:58.056640 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 04:24:58.057876 (+ 1236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:24:58.058408 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:24:58.058416 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:24:58.058801 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:24:58.059377 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:24:58.059394 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:24:58.060239 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:24:58.060242 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:24:58.061134 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:24:58.061145 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:24:58.061654 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:24:58.061659 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:24:58.061821 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:24:58.062438 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:24:58.062455 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:24:58.063917 (+ 1462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:24:58.065816 (+ 1899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:24:58.065821 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:24:58.065834 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:24:58.066104 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:24:58.066374 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:24:58.066376 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:24:58.066377 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:24:58.066509 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:24:58.066899 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:24:58.066904 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:24:58.067148 (+ 244us) client_negotiation.cc:769] Sending connection context
0620 04:24:58.067298 (+ 150us) client_negotiation.cc:240] Negotiation successful
0620 04:24:58.067546 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":156,"threads_started":1}
I20260620 04:24:58.069612 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39111
I20260620 04:24:58.069832 769 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260620 04:24:58.070427 769 heartbeater.cc:511] Master 127.0.98.62:39111 requested a full tablet report, sending...
I20260620 04:24:58.071436 444 ts_manager.cc:194] Registered new tserver with Master: 3c6aad3108aa4199bbaad1f460879176 (127.0.98.2:40407)
I20260620 04:24:58.072214 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:34563
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929496330398-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929496330398-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 04:24:58.084175 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929496330398-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929496330398-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929496330398-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929496330398-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39111
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929496330398-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:24:58.193955 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:24:58.194223 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:24:58.194279 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:24:58.198278 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:24:58.198343 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:24:58.198436 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 04:24:58.203773 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929496330398-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929496330398-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929496330398-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929496330398-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39111
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929496330398-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:24:58.204845 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:24:58.205844 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:24:58.214905 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:24:58.214895 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:24:58.214913 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:24:58.215179 774 server_base.cc:1034] running on GCE node
I20260620 04:24:58.215546 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:24:58.216185 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:24:58.217341 774 hybrid_clock.cc:648] HybridClock initialized: now 1781929498217302 us; error 56 us; skew 500 ppm
Jun 20 04:24:58 dist-test-slave-nk6d krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929498, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:24:58.220031 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 04:24:58.220844 774 webserver.cc:460] Webserver started at http://127.0.98.3:43195/ using document root <none> and password file <none>
I20260620 04:24:58.221340 774 fs_manager.cc:359] Metadata directory not provided
I20260620 04:24:58.221393 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:24:58.221594 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:24:58.223646 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929496330398-392-0/minicluster-data/ts-2/data/instance:
uuid: "63387d48741b4b40ac7fb4cf71479633"
format_stamp: "Formatted at 2026-06-20 04:24:58 on dist-test-slave-nk6d"
server_key: "6528fb7cfd57d08ec2cc945460cd0275"
server_key_iv: "571395ee5397f5cda50304045f8bd893"
server_key_version: "encryptionkey@0"
I20260620 04:24:58.224174 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929496330398-392-0/minicluster-data/ts-2/wal/instance:
uuid: "63387d48741b4b40ac7fb4cf71479633"
format_stamp: "Formatted at 2026-06-20 04:24:58 on dist-test-slave-nk6d"
server_key: "6528fb7cfd57d08ec2cc945460cd0275"
server_key_iv: "571395ee5397f5cda50304045f8bd893"
server_key_version: "encryptionkey@0"
I20260620 04:24:58.228137 774 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 04:24:58.230458 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:24:58.231695 774 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:24:58.231796 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929496330398-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929496330398-392-0/minicluster-data/ts-2/wal
uuid: "63387d48741b4b40ac7fb4cf71479633"
format_stamp: "Formatted at 2026-06-20 04:24:58 on dist-test-slave-nk6d"
server_key: "6528fb7cfd57d08ec2cc945460cd0275"
server_key_iv: "571395ee5397f5cda50304045f8bd893"
server_key_version: "encryptionkey@0"
I20260620 04:24:58.231885 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929496330398-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929496330398-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929496330398-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:24:58.247226 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:24:58.250329 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:24:58.250483 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:24:58.251046 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:24:58.251813 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:24:58.251857 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:24:58.251896 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:24:58.251922 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:24:58.260690 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41933
I20260620 04:24:58.260782 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41933 every 8 connection(s)
I20260620 04:24:58.261619 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929496330398-392-0/minicluster-data/ts-2/data/info.pb
I20260620 04:24:58.269640 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 774
I20260620 04:24:58.269760 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929496330398-392-0/minicluster-data/ts-2/wal/instance
I20260620 04:24:58.270020 392 external_mini_cluster.cc:1428] Setting key 4f02d156d77dfaa4e8e6be7e4ae7285f
Jun 20 04:24:58 dist-test-slave-nk6d krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929498, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:24:58.275624 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:24:58.263355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59447 (local address 127.0.98.62:39111)
0620 04:24:58.263494 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 04:24:58.263497 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:24:58.264748 (+ 1251us) server_negotiation.cc:407] Connection header received
0620 04:24:58.266038 (+ 1290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:24:58.266041 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:24:58.266093 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:24:58.266222 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:24:58.267321 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:24:58.267865 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:24:58.269007 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:24:58.269226 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:24:58.269332 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:24:58.272245 (+ 2913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:24:58.272248 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:24:58.272250 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:24:58.272274 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:24:58.273710 (+ 1436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:24:58.274219 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:24:58.274221 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:24:58.274222 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:24:58.274294 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:24:58.274634 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:24:58.274636 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:24:58.274637 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:24:58.274812 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:24:58.274883 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 04:24:58.275405 (+ 522us) server_negotiation.cc:299] Negotiation successful
0620 04:24:58.275509 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 04:24:58.276494 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:24:58.263757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39111 (local address 127.0.98.3:59447)
0620 04:24:58.264215 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0620 04:24:58.264256 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 04:24:58.265817 (+ 1561us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:24:58.266336 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:24:58.266343 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:24:58.266690 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:24:58.267183 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:24:58.267196 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:24:58.268000 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:24:58.268004 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:24:58.268872 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:24:58.268880 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:24:58.269358 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:24:58.269360 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:24:58.269550 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:24:58.270315 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:24:58.270335 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:24:58.272076 (+ 1741us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:24:58.273838 (+ 1762us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:24:58.273843 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:24:58.273855 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:24:58.274130 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:24:58.274387 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:24:58.274389 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:24:58.274391 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:24:58.274540 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:24:58.274889 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:24:58.274895 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:24:58.275181 (+ 286us) client_negotiation.cc:769] Sending connection context
0620 04:24:58.275332 (+ 151us) client_negotiation.cc:240] Negotiation successful
0620 04:24:58.275592 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":138,"threads_started":1}
I20260620 04:24:58.277686 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39111
I20260620 04:24:58.277915 903 heartbeater.cc:464] Registering TS with master...
I20260620 04:24:58.278344 903 heartbeater.cc:511] Master 127.0.98.62:39111 requested a full tablet report, sending...
I20260620 04:24:58.279260 444 ts_manager.cc:194] Registered new tserver with Master: 63387d48741b4b40ac7fb4cf71479633 (127.0.98.3:41933)
I20260620 04:24:58.279767 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59447
I20260620 04:24:58.284665 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 04:24:58 dist-test-slave-nk6d krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929498, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 04:24:58 dist-test-slave-nk6d krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929498, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 04:24:58.308759 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:24:58.300066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55566 (local address 127.0.98.1:40273)
0620 04:24:58.300392 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0620 04:24:58.300402 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 04:24:58.300429 (+ 27us) server_negotiation.cc:407] Connection header received
0620 04:24:58.300544 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:24:58.300549 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:24:58.300740 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:24:58.300861 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:24:58.301510 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:24:58.302270 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:24:58.303073 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:24:58.303445 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:24:58.303535 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:24:58.305870 (+ 2335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:24:58.305879 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:24:58.305893 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:24:58.305916 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:24:58.307276 (+ 1360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:24:58.307656 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:24:58.307661 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:24:58.307662 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:24:58.307735 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:24:58.307915 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:24:58.307917 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:24:58.307918 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:24:58.308174 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:24:58.308342 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0620 04:24:58.308488 (+ 146us) server_negotiation.cc:299] Negotiation successful
0620 04:24:58.308608 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":133,"threads_started":1}
W20260620 04:24:58.309708 549 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55566
Jun 20 04:24:58 dist-test-slave-nk6d krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929498, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 04:24:58 dist-test-slave-nk6d krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929498, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 04:24:58.333804 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:24:58.325649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55568 (local address 127.0.98.1:40273)
0620 04:24:58.325782 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 04:24:58.325786 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:24:58.326369 (+ 583us) server_negotiation.cc:407] Connection header received
0620 04:24:58.326532 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:24:58.326535 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:24:58.326580 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:24:58.326644 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:24:58.327289 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:24:58.327827 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:24:58.328645 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:24:58.328868 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:24:58.328939 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:24:58.331119 (+ 2180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:24:58.331122 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:24:58.331124 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:24:58.331149 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:24:58.332545 (+ 1396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:24:58.332906 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:24:58.332908 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:24:58.332909 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:24:58.332978 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:24:58.333158 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:24:58.333160 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:24:58.333161 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:24:58.333316 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:24:58.333404 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 04:24:58.333593 (+ 189us) server_negotiation.cc:299] Negotiation successful
0620 04:24:58.333685 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 04:24:58.334722 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 506
I20260620 04:24:58.340117 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 640
I20260620 04:24:58.346596 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 774
I20260620 04:24:58.351034 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 412
2026-06-20T04:24:58Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2014 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929496330398-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:24:58 dist-test-slave-nk6d krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:24:58 dist-test-slave-nk6d krb5kdc[917](info): set up 2 sockets
Jun 20 04:24:58 dist-test-slave-nk6d krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260620 04:24:59.299446 392 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929496330398-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929496330398-392-0/krb5kdc/test-user.keytab.
Jun 20 04:24:59 dist-test-slave-nk6d krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929499, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:24:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:24:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 04:24:59.410921 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929496330398-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929496330398-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929496330398-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41029
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41029
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:24:59.517925 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:24:59.518246 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:24:59.518298 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:24:59.522449 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:24:59.522523 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:24:59.522542 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:24:59.522563 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:24:59.522580 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:24:59.527976 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929496330398-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929496330398-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41029
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41029
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929496330398-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:24:59.529022 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:24:59.530051 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:24:59.537952 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:24:59.537987 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:24:59.537961 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:24:59.538144 930 server_base.cc:1034] running on GCE node
I20260620 04:24:59.538787 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:24:59.539848 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:24:59.541008 930 hybrid_clock.cc:648] HybridClock initialized: now 1781929499540986 us; error 37 us; skew 500 ppm
Jun 20 04:24:59 dist-test-slave-nk6d krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929499, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:24:59.544677 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 04:24:59.545769 930 webserver.cc:460] Webserver started at http://127.0.98.62:42303/ using document root <none> and password file <none>
I20260620 04:24:59.546411 930 fs_manager.cc:359] Metadata directory not provided
I20260620 04:24:59.546502 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:24:59.546808 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:24:59.548916 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929496330398-392-0/minicluster-data/master-0/data/instance:
uuid: "ce177201b34849b996bfcf21a27a413e"
format_stamp: "Formatted at 2026-06-20 04:24:59 on dist-test-slave-nk6d"
server_key: "a0d0d34a17c041598d169906f2d13365"
server_key_iv: "04fbb729110954bc856dbae3903a5ffd"
server_key_version: "encryptionkey@0"
I20260620 04:24:59.549453 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929496330398-392-0/minicluster-data/master-0/wal/instance:
uuid: "ce177201b34849b996bfcf21a27a413e"
format_stamp: "Formatted at 2026-06-20 04:24:59 on dist-test-slave-nk6d"
server_key: "a0d0d34a17c041598d169906f2d13365"
server_key_iv: "04fbb729110954bc856dbae3903a5ffd"
server_key_version: "encryptionkey@0"
I20260620 04:24:59.553098 930 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260620 04:24:59.555409 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:24:59.556665 930 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:24:59.556777 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929496330398-392-0/minicluster-data/master-0/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929496330398-392-0/minicluster-data/master-0/wal
uuid: "ce177201b34849b996bfcf21a27a413e"
format_stamp: "Formatted at 2026-06-20 04:24:59 on dist-test-slave-nk6d"
server_key: "a0d0d34a17c041598d169906f2d13365"
server_key_iv: "04fbb729110954bc856dbae3903a5ffd"
server_key_version: "encryptionkey@0"
I20260620 04:24:59.556869 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929496330398-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929496330398-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929496330398-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:24:59.586161 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:24:59.589208 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:24:59.589412 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:24:59.595925 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41029
I20260620 04:24:59.595989 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41029 every 8 connection(s)
I20260620 04:24:59.596907 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:24:59.599622 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:24:59.605103 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce177201b34849b996bfcf21a27a413e: Bootstrap starting.
I20260620 04:24:59.606694 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 930
I20260620 04:24:59.606798 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929496330398-392-0/minicluster-data/master-0/wal/instance
I20260620 04:24:59.607049 392 external_mini_cluster.cc:1428] Setting key 8afaf9603dea6b73a73cb32cd8fb194f
I20260620 04:24:59.607936 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ce177201b34849b996bfcf21a27a413e: Neither blocks nor log segments found. Creating new log.
I20260620 04:24:59.608794 998 log.cc:824] T 00000000000000000000000000000000 P ce177201b34849b996bfcf21a27a413e: Log is configured to *not* fsync() on all Append() calls
I20260620 04:24:59.610744 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce177201b34849b996bfcf21a27a413e: No bootstrap required, opened a new log
I20260620 04:24:59.614195 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P ce177201b34849b996bfcf21a27a413e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce177201b34849b996bfcf21a27a413e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41029 } }
I20260620 04:24:59.614405 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P ce177201b34849b996bfcf21a27a413e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:24:59.614446 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P ce177201b34849b996bfcf21a27a413e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce177201b34849b996bfcf21a27a413e, State: Initialized, Role: FOLLOWER
I20260620 04:24:59.614923 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P ce177201b34849b996bfcf21a27a413e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce177201b34849b996bfcf21a27a413e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41029 } }
I20260620 04:24:59.615082 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P ce177201b34849b996bfcf21a27a413e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:24:59.615149 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P ce177201b34849b996bfcf21a27a413e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:24:59.615242 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ce177201b34849b996bfcf21a27a413e [term 0 FOLLOWER]: Advancing to term 1
Jun 20 04:24:59 dist-test-slave-nk6d krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929499, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:24:59.616248 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P ce177201b34849b996bfcf21a27a413e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce177201b34849b996bfcf21a27a413e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41029 } }
I20260620 04:24:59.616631 998 leader_election.cc:302] T 00000000000000000000000000000000 P ce177201b34849b996bfcf21a27a413e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ce177201b34849b996bfcf21a27a413e; no voters:
I20260620 04:24:59.616948 998 leader_election.cc:288] T 00000000000000000000000000000000 P ce177201b34849b996bfcf21a27a413e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:24:59.617033 1003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ce177201b34849b996bfcf21a27a413e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:24:59.617215 1003 raft_consensus.cc:686] T 00000000000000000000000000000000 P ce177201b34849b996bfcf21a27a413e [term 1 LEADER]: Becoming Leader. State: Replica: ce177201b34849b996bfcf21a27a413e, State: Running, Role: LEADER
I20260620 04:24:59.617560 1003 consensus_queue.cc:237] T 00000000000000000000000000000000 P ce177201b34849b996bfcf21a27a413e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce177201b34849b996bfcf21a27a413e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41029 } }
I20260620 04:24:59.618185 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P ce177201b34849b996bfcf21a27a413e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:24:59.619091 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce177201b34849b996bfcf21a27a413e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ce177201b34849b996bfcf21a27a413e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce177201b34849b996bfcf21a27a413e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41029 } } }
I20260620 04:24:59.619295 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce177201b34849b996bfcf21a27a413e [sys.catalog]: This master's current role is: LEADER
I20260620 04:24:59.619545 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce177201b34849b996bfcf21a27a413e [sys.catalog]: SysCatalogTable state changed. Reason: New leader ce177201b34849b996bfcf21a27a413e. Latest consensus state: current_term: 1 leader_uuid: "ce177201b34849b996bfcf21a27a413e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce177201b34849b996bfcf21a27a413e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41029 } } }
I20260620 04:24:59.619688 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce177201b34849b996bfcf21a27a413e [sys.catalog]: This master's current role is: LEADER
I20260620 04:24:59.621385 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:24:59.608220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32932 (local address 127.0.98.62:41029)
0620 04:24:59.608665 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0620 04:24:59.608681 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 04:24:59.608716 (+ 35us) server_negotiation.cc:407] Connection header received
0620 04:24:59.609437 (+ 721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:24:59.609462 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:24:59.609831 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:24:59.610143 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:24:59.610891 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:24:59.611844 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:24:59.612687 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:24:59.613145 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:24:59.613298 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:24:59.615834 (+ 2536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:24:59.615841 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:24:59.615858 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:24:59.615888 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:24:59.617682 (+ 1794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:24:59.618062 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:24:59.618068 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:24:59.618074 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:24:59.618203 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:24:59.618447 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:24:59.618450 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:24:59.618452 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:24:59.618928 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:24:59.619754 (+ 826us) server_negotiation.cc:1035] Waiting for connection context
0620 04:24:59.620384 (+ 630us) server_negotiation.cc:299] Negotiation successful
0620 04:24:59.620679 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":118,"threads_started":1}
W20260620 04:24:59.622823 1018 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ce177201b34849b996bfcf21a27a413e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 04:24:59.622895 1018 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 04:24:59.622991 1012 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:24:59.623838 1012 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:24:59.629730 1012 catalog_manager.cc:1347] Generated new cluster ID: e649b0b3164b448dbb41e344c6c94bf4
I20260620 04:24:59.629803 1012 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:24:59.646507 1012 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:24:59.647775 1012 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:24:59.655189 1012 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ce177201b34849b996bfcf21a27a413e: Generated new TSK 0
I20260620 04:24:59.656083 1012 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929496330398-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929496330398-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 04:24:59.686775 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929496330398-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929496330398-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929496330398-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929496330398-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41029
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929496330398-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:24:59.810084 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:24:59.810344 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:24:59.810388 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:24:59.814332 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:24:59.814384 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:24:59.814453 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 04:24:59.819756 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929496330398-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929496330398-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929496330398-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929496330398-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41029
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929496330398-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:24:59.820773 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:24:59.821760 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:24:59.831104 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:24:59.831204 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:24:59.831280 1024 server_base.cc:1034] running on GCE node
W20260620 04:24:59.832000 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:24:59.832387 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:24:59.833011 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:24:59.834173 1024 hybrid_clock.cc:648] HybridClock initialized: now 1781929499834156 us; error 36 us; skew 500 ppm
Jun 20 04:24:59 dist-test-slave-nk6d krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929499, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:24:59.836977 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 04:24:59.837832 1024 webserver.cc:460] Webserver started at http://127.0.98.1:34789/ using document root <none> and password file <none>
I20260620 04:24:59.838367 1024 fs_manager.cc:359] Metadata directory not provided
I20260620 04:24:59.838430 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:24:59.838624 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:24:59.840673 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929496330398-392-0/minicluster-data/ts-0/data/instance:
uuid: "9374a3b0ad334e898b753c0157b127ee"
format_stamp: "Formatted at 2026-06-20 04:24:59 on dist-test-slave-nk6d"
server_key: "852ed63fc20fcbfe70626f4b70ae573d"
server_key_iv: "246fe642e41e0ffb70ccda8adcc71672"
server_key_version: "encryptionkey@0"
I20260620 04:24:59.841256 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929496330398-392-0/minicluster-data/ts-0/wal/instance:
uuid: "9374a3b0ad334e898b753c0157b127ee"
format_stamp: "Formatted at 2026-06-20 04:24:59 on dist-test-slave-nk6d"
server_key: "852ed63fc20fcbfe70626f4b70ae573d"
server_key_iv: "246fe642e41e0ffb70ccda8adcc71672"
server_key_version: "encryptionkey@0"
I20260620 04:24:59.844944 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 04:24:59.847299 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:24:59.848516 1024 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:24:59.848632 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929496330398-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929496330398-392-0/minicluster-data/ts-0/wal
uuid: "9374a3b0ad334e898b753c0157b127ee"
format_stamp: "Formatted at 2026-06-20 04:24:59 on dist-test-slave-nk6d"
server_key: "852ed63fc20fcbfe70626f4b70ae573d"
server_key_iv: "246fe642e41e0ffb70ccda8adcc71672"
server_key_version: "encryptionkey@0"
I20260620 04:24:59.848721 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929496330398-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929496330398-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929496330398-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:24:59.861766 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:24:59.864785 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:24:59.864957 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:24:59.865516 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:24:59.866290 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:24:59.866335 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:24:59.866387 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:24:59.866413 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:24:59.874531 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35069
I20260620 04:24:59.874552 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35069 every 8 connection(s)
I20260620 04:24:59.875353 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929496330398-392-0/minicluster-data/ts-0/data/info.pb
I20260620 04:24:59.882987 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 1024
I20260620 04:24:59.883097 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929496330398-392-0/minicluster-data/ts-0/wal/instance
I20260620 04:24:59.883350 392 external_mini_cluster.cc:1428] Setting key af04fc15e825e1d45a4845615a847d17
Jun 20 04:24:59 dist-test-slave-nk6d krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929499, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:24:59.888870 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:24:59.877078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50189 (local address 127.0.98.62:41029)
0620 04:24:59.877225 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 04:24:59.877229 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:24:59.878017 (+ 788us) server_negotiation.cc:407] Connection header received
0620 04:24:59.879438 (+ 1421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:24:59.879441 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:24:59.879488 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:24:59.879597 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:24:59.880818 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:24:59.881524 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:24:59.882593 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:24:59.882883 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:24:59.882989 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:24:59.885583 (+ 2594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:24:59.885586 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:24:59.885588 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:24:59.885609 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:24:59.886931 (+ 1322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:24:59.887451 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:24:59.887453 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:24:59.887454 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:24:59.887516 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:24:59.887842 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:24:59.887844 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:24:59.887845 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:24:59.888035 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:24:59.888104 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 04:24:59.888660 (+ 556us) server_negotiation.cc:299] Negotiation successful
0620 04:24:59.888755 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 04:24:59.889988 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:24:59.877428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41029 (local address 127.0.98.1:50189)
0620 04:24:59.877867 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0620 04:24:59.877908 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 04:24:59.879213 (+ 1305us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:24:59.879752 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:24:59.879758 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:24:59.880174 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:24:59.880638 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:24:59.880650 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:24:59.881625 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:24:59.881627 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:24:59.882484 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:24:59.882492 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:24:59.883012 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:24:59.883014 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:24:59.883183 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:24:59.883780 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:24:59.883798 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:24:59.885433 (+ 1635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:24:59.887037 (+ 1604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:24:59.887042 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:24:59.887057 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:24:59.887369 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:24:59.887586 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:24:59.887589 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:24:59.887591 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:24:59.887761 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:24:59.888120 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:24:59.888126 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:24:59.888423 (+ 297us) client_negotiation.cc:769] Sending connection context
0620 04:24:59.888631 (+ 208us) client_negotiation.cc:240] Negotiation successful
0620 04:24:59.888964 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":120,"threads_started":1}
I20260620 04:24:59.891136 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41029
I20260620 04:24:59.891363 1153 heartbeater.cc:464] Registering TS with master...
I20260620 04:24:59.891819 1153 heartbeater.cc:511] Master 127.0.98.62:41029 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 04:24:59.893299 962 ts_manager.cc:194] Registered new tserver with Master: 9374a3b0ad334e898b753c0157b127ee (127.0.98.1:35069)
I20260620 04:24:59.894428 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:50189
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929496330398-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929496330398-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 04:24:59.907745 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929496330398-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929496330398-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929496330398-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929496330398-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41029
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929496330398-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:25:00.019793 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:25:00.020135 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:25:00.020196 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:25:00.024478 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:25:00.024554 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:25:00.024644 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 04:25:00.030284 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929496330398-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929496330398-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929496330398-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929496330398-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41029
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929496330398-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:25:00.031386 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:25:00.032418 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:25:00.041885 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:25:00.042008 1158 server_base.cc:1034] running on GCE node
W20260620 04:25:00.041901 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:25:00.041888 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:25:00.042600 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:25:00.043246 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:25:00.044431 1158 hybrid_clock.cc:648] HybridClock initialized: now 1781929500044399 us; error 46 us; skew 500 ppm
Jun 20 04:25:00 dist-test-slave-nk6d krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929500, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:25:00.047204 1158 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 04:25:00.048105 1158 webserver.cc:460] Webserver started at http://127.0.98.2:45879/ using document root <none> and password file <none>
I20260620 04:25:00.048707 1158 fs_manager.cc:359] Metadata directory not provided
I20260620 04:25:00.048791 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:25:00.048997 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:25:00.051513 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929496330398-392-0/minicluster-data/ts-1/data/instance:
uuid: "749ce7bc297846778acfda0c2b755f2e"
format_stamp: "Formatted at 2026-06-20 04:25:00 on dist-test-slave-nk6d"
server_key: "3dbd517eec59b23d7da40c7d9401fd18"
server_key_iv: "d7d8b96b743ce0510ba887f6a105830e"
server_key_version: "encryptionkey@0"
I20260620 04:25:00.052217 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929496330398-392-0/minicluster-data/ts-1/wal/instance:
uuid: "749ce7bc297846778acfda0c2b755f2e"
format_stamp: "Formatted at 2026-06-20 04:25:00 on dist-test-slave-nk6d"
server_key: "3dbd517eec59b23d7da40c7d9401fd18"
server_key_iv: "d7d8b96b743ce0510ba887f6a105830e"
server_key_version: "encryptionkey@0"
I20260620 04:25:00.056237 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260620 04:25:00.058799 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:00.060150 1158 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:25:00.060245 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929496330398-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929496330398-392-0/minicluster-data/ts-1/wal
uuid: "749ce7bc297846778acfda0c2b755f2e"
format_stamp: "Formatted at 2026-06-20 04:25:00 on dist-test-slave-nk6d"
server_key: "3dbd517eec59b23d7da40c7d9401fd18"
server_key_iv: "d7d8b96b743ce0510ba887f6a105830e"
server_key_version: "encryptionkey@0"
I20260620 04:25:00.060333 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929496330398-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929496330398-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929496330398-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:25:00.085575 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:25:00.088856 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:25:00.089020 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:25:00.089658 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:25:00.090461 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:25:00.090507 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:00.090545 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:25:00.090565 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:00.098934 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46225
I20260620 04:25:00.099004 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46225 every 8 connection(s)
I20260620 04:25:00.099838 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929496330398-392-0/minicluster-data/ts-1/data/info.pb
I20260620 04:25:00.103729 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 1158
I20260620 04:25:00.103837 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929496330398-392-0/minicluster-data/ts-1/wal/instance
I20260620 04:25:00.104090 392 external_mini_cluster.cc:1428] Setting key 17977b54c6739817578e2657be2bd732
Jun 20 04:25:00 dist-test-slave-nk6d krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929500, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 04:25:00.114107 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:00.102401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44233 (local address 127.0.98.62:41029)
0620 04:25:00.102572 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:00.102576 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:00.103441 (+ 865us) server_negotiation.cc:407] Connection header received
0620 04:25:00.104845 (+ 1404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:00.104848 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:00.104897 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:00.104967 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:00.106190 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:00.106699 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:00.107839 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:00.108092 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:00.108176 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:00.110743 (+ 2567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:00.110747 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:00.110750 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:25:00.110782 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:00.112206 (+ 1424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:00.112676 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:00.112679 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:00.112681 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:00.112756 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:00.113056 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:00.113059 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:00.113061 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:00.113241 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:00.113323 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:00.113819 (+ 496us) server_negotiation.cc:299] Negotiation successful
0620 04:25:00.113947 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 04:25:00.114934 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:00.102811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41029 (local address 127.0.98.2:44233)
0620 04:25:00.103290 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:00.103334 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:00.104638 (+ 1304us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:00.105156 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:00.105162 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:00.105569 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:00.106051 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:00.106064 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:00.106818 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:00.106821 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:00.107725 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:00.107734 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:00.108239 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:00.108243 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:00.108423 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:00.108971 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:00.108986 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:00.110579 (+ 1593us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:25:00.112313 (+ 1734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:00.112318 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:00.112340 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:00.112584 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:00.112835 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:00.112838 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:00.112840 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:00.112973 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:00.113335 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:25:00.113342 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:25:00.113597 (+ 255us) client_negotiation.cc:769] Sending connection context
0620 04:25:00.113768 (+ 171us) client_negotiation.cc:240] Negotiation successful
0620 04:25:00.114107 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":122,"threads_started":1}
I20260620 04:25:00.116192 1287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41029
I20260620 04:25:00.116447 1287 heartbeater.cc:464] Registering TS with master...
I20260620 04:25:00.116955 1287 heartbeater.cc:511] Master 127.0.98.62:41029 requested a full tablet report, sending...
I20260620 04:25:00.117990 962 ts_manager.cc:194] Registered new tserver with Master: 749ce7bc297846778acfda0c2b755f2e (127.0.98.2:46225)
I20260620 04:25:00.118584 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:44233
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260620 04:25:00.896824 1153 heartbeater.cc:503] Master 127.0.98.62:41029 was elected leader, sending a full tablet report...
I20260620 04:25:01.120940 1287 heartbeater.cc:503] Master 127.0.98.62:41029 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929496330398-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929496330398-392-0/krb5kdc/kudu_127.0.98.3.keytab.
W20260620 04:25:01.471998 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.3: real 1.366s user 0.002s sys 0.001s
I20260620 04:25:01.472479 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929496330398-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929496330398-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929496330398-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929496330398-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41029
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929496330398-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:25:01.587527 1292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:25:01.587839 1292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:25:01.587885 1292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:25:01.591923 1292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:25:01.592023 1292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:25:01.592089 1292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 04:25:01.597510 1292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929496330398-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929496330398-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929496330398-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929496330398-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41029
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929496330398-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:25:01.598693 1292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:25:01.599730 1292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 04:25:01.609575 1292 server_base.cc:1034] running on GCE node
W20260620 04:25:01.609586 1297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:25:01.609570 1298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:25:01.609560 1300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:25:01.610258 1292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:25:01.610885 1292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:25:01.612044 1292 hybrid_clock.cc:648] HybridClock initialized: now 1781929501612017 us; error 40 us; skew 500 ppm
Jun 20 04:25:01 dist-test-slave-nk6d krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929501, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:25:01.614934 1292 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 04:25:01.615834 1292 webserver.cc:460] Webserver started at http://127.0.98.3:38379/ using document root <none> and password file <none>
I20260620 04:25:01.616443 1292 fs_manager.cc:359] Metadata directory not provided
I20260620 04:25:01.616513 1292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:25:01.616701 1292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:25:01.619029 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929496330398-392-0/minicluster-data/ts-2/data/instance:
uuid: "efcf273e609247c8bf3b24dd56cf78e5"
format_stamp: "Formatted at 2026-06-20 04:25:01 on dist-test-slave-nk6d"
server_key: "10893ed154c56809580434ed62907c45"
server_key_iv: "377c27e1816a35e721ce53f00276d077"
server_key_version: "encryptionkey@0"
I20260620 04:25:01.619629 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929496330398-392-0/minicluster-data/ts-2/wal/instance:
uuid: "efcf273e609247c8bf3b24dd56cf78e5"
format_stamp: "Formatted at 2026-06-20 04:25:01 on dist-test-slave-nk6d"
server_key: "10893ed154c56809580434ed62907c45"
server_key_iv: "377c27e1816a35e721ce53f00276d077"
server_key_version: "encryptionkey@0"
I20260620 04:25:01.623664 1292 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260620 04:25:01.626354 1307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:01.627687 1292 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 04:25:01.627826 1292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929496330398-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929496330398-392-0/minicluster-data/ts-2/wal
uuid: "efcf273e609247c8bf3b24dd56cf78e5"
format_stamp: "Formatted at 2026-06-20 04:25:01 on dist-test-slave-nk6d"
server_key: "10893ed154c56809580434ed62907c45"
server_key_iv: "377c27e1816a35e721ce53f00276d077"
server_key_version: "encryptionkey@0"
I20260620 04:25:01.627926 1292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929496330398-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929496330398-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929496330398-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:25:01.643867 1292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:25:01.647706 1292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:25:01.647924 1292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:25:01.648655 1292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:25:01.649505 1292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:25:01.649550 1292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:01.649600 1292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:25:01.649628 1292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:01.659507 1292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34219
I20260620 04:25:01.659605 1420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34219 every 8 connection(s)
I20260620 04:25:01.660496 1292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929496330398-392-0/minicluster-data/ts-2/data/info.pb
I20260620 04:25:01.669004 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 1292
I20260620 04:25:01.669106 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929496330398-392-0/minicluster-data/ts-2/wal/instance
I20260620 04:25:01.669333 392 external_mini_cluster.cc:1428] Setting key 3aa314fb7eef4223722e1ec748ba566f
Jun 20 04:25:01 dist-test-slave-nk6d krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929501, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:25:01.675073 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:01.662306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59647 (local address 127.0.98.62:41029)
0620 04:25:01.662546 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:01.662549 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:01.663279 (+ 730us) server_negotiation.cc:407] Connection header received
0620 04:25:01.664771 (+ 1492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:01.664774 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:01.664824 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:01.664905 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:01.666075 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:01.666638 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:01.667733 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:01.667992 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:01.668084 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:01.670762 (+ 2678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:01.670766 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:01.670769 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:25:01.670818 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:01.672714 (+ 1896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:01.673334 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:01.673338 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:01.673340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:01.673443 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:01.673820 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:01.673823 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:01.673825 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:01.674053 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:01.674143 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:01.674771 (+ 628us) server_negotiation.cc:299] Negotiation successful
0620 04:25:01.674908 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":79,"threads_started":1}
I20260620 04:25:01.676271 1424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:01.662717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41029 (local address 127.0.98.3:59647)
0620 04:25:01.663147 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:01.663187 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:01.664547 (+ 1360us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:01.665070 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:01.665077 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:01.665437 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:01.665940 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:01.665954 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:01.666751 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:01.666754 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:01.667637 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:01.667645 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:01.668153 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:01.668156 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:01.668315 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:01.668854 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:01.668873 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:01.670560 (+ 1687us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:25:01.672826 (+ 2266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:01.672833 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:01.672852 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:01.673240 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:01.673528 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:01.673531 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:01.673533 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:01.673726 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:01.674150 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:25:01.674157 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:25:01.674519 (+ 362us) client_negotiation.cc:769] Sending connection context
0620 04:25:01.674742 (+ 223us) client_negotiation.cc:240] Negotiation successful
0620 04:25:01.675124 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":114,"threads_started":1}
I20260620 04:25:01.677489 1421 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41029
I20260620 04:25:01.677729 1421 heartbeater.cc:464] Registering TS with master...
I20260620 04:25:01.678262 1421 heartbeater.cc:511] Master 127.0.98.62:41029 requested a full tablet report, sending...
I20260620 04:25:01.679363 962 ts_manager.cc:194] Registered new tserver with Master: efcf273e609247c8bf3b24dd56cf78e5 (127.0.98.3:34219)
I20260620 04:25:01.680078 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59647
I20260620 04:25:01.683519 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 04:25:01.694128 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:01.686882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37232 (local address 127.0.98.62:41029)
0620 04:25:01.687019 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:01.687024 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:01.687177 (+ 153us) server_negotiation.cc:407] Connection header received
0620 04:25:01.687252 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:01.687254 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:01.687314 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:01.687382 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:01.688082 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:01.688623 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:01.689418 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:01.689634 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:01.689709 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:01.691025 (+ 1316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:01.691029 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:01.691031 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:25:01.691063 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:01.692527 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:01.692946 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:01.692949 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:01.692951 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:01.693038 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:01.693339 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:01.693342 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:01.693344 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:01.693573 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:01.693681 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:01.693869 (+ 188us) server_negotiation.cc:299] Negotiation successful
0620 04:25:01.693976 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 04:25:01.699051 962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37232:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 04:25:01.702035 962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 04:25:01.717836 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:01.710849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46225 (local address 127.0.0.1:39664)
0620 04:25:01.711852 (+ 1003us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:01.711864 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:01.711952 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:01.712608 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:01.712611 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:01.712624 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:25:01.712688 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:01.712693 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:01.714198 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:01.714202 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:01.715864 (+ 1662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:01.715874 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:01.717372 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:01.717376 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:01.717455 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:01.717478 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 04:25:01.717539 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 04:25:01.717668 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":938,"mutex_wait_us":192,"thread_start_us":93,"threads_started":1}
I20260620 04:25:01.718370 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:01.710657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35069 (local address 127.0.0.1:36200)
0620 04:25:01.711342 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:01.711398 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:01.711523 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:01.712490 (+ 967us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:01.712496 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:01.712517 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:25:01.712638 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:01.712645 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:01.714198 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:01.714202 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:01.715863 (+ 1661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:01.715874 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:01.717022 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:01.717026 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:01.717146 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:01.717191 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 04:25:01.717289 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 04:25:01.718219 (+ 930us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"mutex_wait_us":85,"thread_start_us":115,"threads_started":1}
I20260620 04:25:01.718454 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:01.711476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39664 (local address 127.0.98.2:46225)
0620 04:25:01.711805 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:01.711812 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:01.712065 (+ 253us) server_negotiation.cc:407] Connection header received
0620 04:25:01.712127 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:01.712131 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:01.712331 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:01.712476 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:25:01.712790 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:01.714058 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:01.716222 (+ 2164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:01.717055 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:01.717156 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:01.717315 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:01.718197 (+ 882us) server_negotiation.cc:299] Negotiation successful
0620 04:25:01.718317 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":107,"threads_started":1}
I20260620 04:25:01.718904 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:01.711021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34219 (local address 127.0.0.1:48842)
0620 04:25:01.711906 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:01.711919 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:01.712044 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:01.712788 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:01.712791 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:01.712807 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:25:01.712891 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:01.712896 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:01.716182 (+ 3286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:01.716185 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:01.717290 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:01.717300 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:01.718493 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:01.718497 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:01.718580 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:01.718604 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 04:25:01.718662 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 04:25:01.718787 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":826,"thread_start_us":86,"threads_started":1}
I20260620 04:25:01.719123 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:01.710816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36200 (local address 127.0.98.1:35069)
0620 04:25:01.712008 (+ 1192us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:01.712014 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:01.712038 (+ 24us) server_negotiation.cc:407] Connection header received
0620 04:25:01.712113 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:01.712119 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:01.712335 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:01.712776 (+ 441us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:25:01.712900 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:01.714083 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:01.716022 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:01.716892 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:01.718672 (+ 1780us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:01.718828 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:01.718881 (+ 53us) server_negotiation.cc:299] Negotiation successful
0620 04:25:01.718983 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1069,"thread_start_us":118,"threads_started":1}
I20260620 04:25:01.719411 1439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:01.711479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48842 (local address 127.0.98.3:34219)
0620 04:25:01.712218 (+ 739us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:01.712228 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:01.712258 (+ 30us) server_negotiation.cc:407] Connection header received
0620 04:25:01.712326 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:01.712331 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:01.712509 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:01.712639 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:25:01.713010 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:01.716064 (+ 3054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:01.717421 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:01.718204 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:01.718298 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:01.719086 (+ 788us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:01.719136 (+ 50us) server_negotiation.cc:299] Negotiation successful
0620 04:25:01.719266 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":660,"thread_start_us":130,"threads_started":1}
I20260620 04:25:01.722028 1221 tablet_service.cc:1467] Processing CreateTablet for tablet a8ae46a5ccab40f7807a83888b174f76 (DEFAULT_TABLE table=test-table [id=d5fbb424d3c049da9eef1607a3cea479]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:25:01.722028 1087 tablet_service.cc:1467] Processing CreateTablet for tablet a8ae46a5ccab40f7807a83888b174f76 (DEFAULT_TABLE table=test-table [id=d5fbb424d3c049da9eef1607a3cea479]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:25:01.722038 1355 tablet_service.cc:1467] Processing CreateTablet for tablet a8ae46a5ccab40f7807a83888b174f76 (DEFAULT_TABLE table=test-table [id=d5fbb424d3c049da9eef1607a3cea479]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:25:01.723039 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8ae46a5ccab40f7807a83888b174f76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:25:01.723053 1355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8ae46a5ccab40f7807a83888b174f76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:25:01.723053 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8ae46a5ccab40f7807a83888b174f76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:25:01.729915 1440 tablet_bootstrap.cc:492] T a8ae46a5ccab40f7807a83888b174f76 P 749ce7bc297846778acfda0c2b755f2e: Bootstrap starting.
I20260620 04:25:01.730674 1442 tablet_bootstrap.cc:492] T a8ae46a5ccab40f7807a83888b174f76 P efcf273e609247c8bf3b24dd56cf78e5: Bootstrap starting.
I20260620 04:25:01.732352 1441 tablet_bootstrap.cc:492] T a8ae46a5ccab40f7807a83888b174f76 P 9374a3b0ad334e898b753c0157b127ee: Bootstrap starting.
I20260620 04:25:01.732586 1440 tablet_bootstrap.cc:654] T a8ae46a5ccab40f7807a83888b174f76 P 749ce7bc297846778acfda0c2b755f2e: Neither blocks nor log segments found. Creating new log.
I20260620 04:25:01.732666 1442 tablet_bootstrap.cc:654] T a8ae46a5ccab40f7807a83888b174f76 P efcf273e609247c8bf3b24dd56cf78e5: Neither blocks nor log segments found. Creating new log.
I20260620 04:25:01.733685 1440 log.cc:824] T a8ae46a5ccab40f7807a83888b174f76 P 749ce7bc297846778acfda0c2b755f2e: Log is configured to *not* fsync() on all Append() calls
I20260620 04:25:01.733880 1442 log.cc:824] T a8ae46a5ccab40f7807a83888b174f76 P efcf273e609247c8bf3b24dd56cf78e5: Log is configured to *not* fsync() on all Append() calls
I20260620 04:25:01.734493 1441 tablet_bootstrap.cc:654] T a8ae46a5ccab40f7807a83888b174f76 P 9374a3b0ad334e898b753c0157b127ee: Neither blocks nor log segments found. Creating new log.
I20260620 04:25:01.735239 1441 log.cc:824] T a8ae46a5ccab40f7807a83888b174f76 P 9374a3b0ad334e898b753c0157b127ee: Log is configured to *not* fsync() on all Append() calls
I20260620 04:25:01.735849 1440 tablet_bootstrap.cc:492] T a8ae46a5ccab40f7807a83888b174f76 P 749ce7bc297846778acfda0c2b755f2e: No bootstrap required, opened a new log
I20260620 04:25:01.736064 1440 ts_tablet_manager.cc:1397] T a8ae46a5ccab40f7807a83888b174f76 P 749ce7bc297846778acfda0c2b755f2e: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 04:25:01.736927 1442 tablet_bootstrap.cc:492] T a8ae46a5ccab40f7807a83888b174f76 P efcf273e609247c8bf3b24dd56cf78e5: No bootstrap required, opened a new log
I20260620 04:25:01.737123 1442 ts_tablet_manager.cc:1397] T a8ae46a5ccab40f7807a83888b174f76 P efcf273e609247c8bf3b24dd56cf78e5: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260620 04:25:01.737382 1441 tablet_bootstrap.cc:492] T a8ae46a5ccab40f7807a83888b174f76 P 9374a3b0ad334e898b753c0157b127ee: No bootstrap required, opened a new log
I20260620 04:25:01.737568 1441 ts_tablet_manager.cc:1397] T a8ae46a5ccab40f7807a83888b174f76 P 9374a3b0ad334e898b753c0157b127ee: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 04:25:01.740362 1442 raft_consensus.cc:348] T a8ae46a5ccab40f7807a83888b174f76 P efcf273e609247c8bf3b24dd56cf78e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9374a3b0ad334e898b753c0157b127ee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35069 } } peers { permanent_uuid: "749ce7bc297846778acfda0c2b755f2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46225 } } peers { permanent_uuid: "efcf273e609247c8bf3b24dd56cf78e5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34219 } }
I20260620 04:25:01.740625 1442 raft_consensus.cc:374] T a8ae46a5ccab40f7807a83888b174f76 P efcf273e609247c8bf3b24dd56cf78e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:25:01.740820 1442 raft_consensus.cc:729] T a8ae46a5ccab40f7807a83888b174f76 P efcf273e609247c8bf3b24dd56cf78e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efcf273e609247c8bf3b24dd56cf78e5, State: Initialized, Role: FOLLOWER
I20260620 04:25:01.740586 1440 raft_consensus.cc:348] T a8ae46a5ccab40f7807a83888b174f76 P 749ce7bc297846778acfda0c2b755f2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9374a3b0ad334e898b753c0157b127ee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35069 } } peers { permanent_uuid: "749ce7bc297846778acfda0c2b755f2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46225 } } peers { permanent_uuid: "efcf273e609247c8bf3b24dd56cf78e5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34219 } }
I20260620 04:25:01.741372 1440 raft_consensus.cc:374] T a8ae46a5ccab40f7807a83888b174f76 P 749ce7bc297846778acfda0c2b755f2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:25:01.741430 1440 raft_consensus.cc:729] T a8ae46a5ccab40f7807a83888b174f76 P 749ce7bc297846778acfda0c2b755f2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 749ce7bc297846778acfda0c2b755f2e, State: Initialized, Role: FOLLOWER
I20260620 04:25:01.741361 1442 consensus_queue.cc:260] T a8ae46a5ccab40f7807a83888b174f76 P efcf273e609247c8bf3b24dd56cf78e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9374a3b0ad334e898b753c0157b127ee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35069 } } peers { permanent_uuid: "749ce7bc297846778acfda0c2b755f2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46225 } } peers { permanent_uuid: "efcf273e609247c8bf3b24dd56cf78e5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34219 } }
I20260620 04:25:01.741882 1440 consensus_queue.cc:260] T a8ae46a5ccab40f7807a83888b174f76 P 749ce7bc297846778acfda0c2b755f2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9374a3b0ad334e898b753c0157b127ee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35069 } } peers { permanent_uuid: "749ce7bc297846778acfda0c2b755f2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46225 } } peers { permanent_uuid: "efcf273e609247c8bf3b24dd56cf78e5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34219 } }
I20260620 04:25:01.742359 1442 ts_tablet_manager.cc:1428] T a8ae46a5ccab40f7807a83888b174f76 P efcf273e609247c8bf3b24dd56cf78e5: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 04:25:01.742673 1421 heartbeater.cc:503] Master 127.0.98.62:41029 was elected leader, sending a full tablet report...
I20260620 04:25:01.742869 1440 ts_tablet_manager.cc:1428] T a8ae46a5ccab40f7807a83888b174f76 P 749ce7bc297846778acfda0c2b755f2e: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260620 04:25:01.742328 1441 raft_consensus.cc:348] T a8ae46a5ccab40f7807a83888b174f76 P 9374a3b0ad334e898b753c0157b127ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9374a3b0ad334e898b753c0157b127ee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35069 } } peers { permanent_uuid: "749ce7bc297846778acfda0c2b755f2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46225 } } peers { permanent_uuid: "efcf273e609247c8bf3b24dd56cf78e5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34219 } }
I20260620 04:25:01.743168 1441 raft_consensus.cc:374] T a8ae46a5ccab40f7807a83888b174f76 P 9374a3b0ad334e898b753c0157b127ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:25:01.743218 1441 raft_consensus.cc:729] T a8ae46a5ccab40f7807a83888b174f76 P 9374a3b0ad334e898b753c0157b127ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9374a3b0ad334e898b753c0157b127ee, State: Initialized, Role: FOLLOWER
I20260620 04:25:01.743641 1441 consensus_queue.cc:260] T a8ae46a5ccab40f7807a83888b174f76 P 9374a3b0ad334e898b753c0157b127ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9374a3b0ad334e898b753c0157b127ee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35069 } } peers { permanent_uuid: "749ce7bc297846778acfda0c2b755f2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46225 } } peers { permanent_uuid: "efcf273e609247c8bf3b24dd56cf78e5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34219 } }
I20260620 04:25:01.745580 1441 ts_tablet_manager.cc:1428] T a8ae46a5ccab40f7807a83888b174f76 P 9374a3b0ad334e898b753c0157b127ee: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
W20260620 04:25:01.851991 1288 tablet.cc:2369] T a8ae46a5ccab40f7807a83888b174f76 P 749ce7bc297846778acfda0c2b755f2e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 04:25:01.877574 1154 tablet.cc:2369] T a8ae46a5ccab40f7807a83888b174f76 P 9374a3b0ad334e898b753c0157b127ee: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 04:25:01.911497 1422 tablet.cc:2369] T a8ae46a5ccab40f7807a83888b174f76 P efcf273e609247c8bf3b24dd56cf78e5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 04:25:01.933710 1447 raft_consensus.cc:482] T a8ae46a5ccab40f7807a83888b174f76 P 749ce7bc297846778acfda0c2b755f2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:25:01.933971 1447 raft_consensus.cc:504] T a8ae46a5ccab40f7807a83888b174f76 P 749ce7bc297846778acfda0c2b755f2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9374a3b0ad334e898b753c0157b127ee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35069 } } peers { permanent_uuid: "749ce7bc297846778acfda0c2b755f2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46225 } } peers { permanent_uuid: "efcf273e609247c8bf3b24dd56cf78e5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34219 } }
I20260620 04:25:01.935108 1447 leader_election.cc:288] T a8ae46a5ccab40f7807a83888b174f76 P 749ce7bc297846778acfda0c2b755f2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9374a3b0ad334e898b753c0157b127ee (127.0.98.1:35069), efcf273e609247c8bf3b24dd56cf78e5 (127.0.98.3:34219)
I20260620 04:25:01.939350 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:01.935487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39849 (local address 127.0.98.1:35069)
0620 04:25:01.935619 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:01.935624 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:01.935881 (+ 257us) server_negotiation.cc:407] Connection header received
0620 04:25:01.936001 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:01.936004 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:01.936050 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:01.936112 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:25:01.936401 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:01.936906 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:01.938086 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:01.938697 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:01.938811 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:01.938880 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:01.939149 (+ 269us) server_negotiation.cc:299] Negotiation successful
0620 04:25:01.939220 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 04:25:01.939350 1439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:01.935492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38399 (local address 127.0.98.3:34219)
0620 04:25:01.935732 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:01.935736 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:01.935756 (+ 20us) server_negotiation.cc:407] Connection header received
0620 04:25:01.936007 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:01.936009 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:01.936049 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:01.936114 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:25:01.936399 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:01.936905 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:01.938138 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:01.938744 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:01.938842 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:01.938893 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:01.939149 (+ 256us) server_negotiation.cc:299] Negotiation successful
0620 04:25:01.939217 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150}
I20260620 04:25:01.939422 1450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:01.935396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35069 (local address 127.0.98.2:39849)
0620 04:25:01.935783 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:01.935796 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:01.935889 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:01.936133 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:01.936136 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:01.936181 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:25:01.936298 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:01.936303 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:01.937009 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:01.937012 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:01.937966 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:01.937975 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:01.938864 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:01.938868 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:01.938983 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:01.939057 (+ 74us) client_negotiation.cc:769] Sending connection context
0620 04:25:01.939135 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 04:25:01.939231 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"mutex_wait_us":101,"spinlock_wait_cycles":3712,"thread_start_us":103,"threads_started":1}
I20260620 04:25:01.939553 1449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:01.935396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34219 (local address 127.0.98.2:38399)
0620 04:25:01.935687 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:01.935700 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:01.935857 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:01.936135 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:01.936138 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:01.936178 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:25:01.936301 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:01.936306 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:01.937007 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:01.937011 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:01.937988 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:01.937998 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:01.938864 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:01.938868 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:01.939034 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:01.939069 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 04:25:01.939135 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 04:25:01.939231 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"mutex_wait_us":111,"spinlock_wait_cycles":39808,"thread_start_us":110,"threads_started":1}
I20260620 04:25:01.940410 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8ae46a5ccab40f7807a83888b174f76" candidate_uuid: "749ce7bc297846778acfda0c2b755f2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efcf273e609247c8bf3b24dd56cf78e5" is_pre_election: true
I20260620 04:25:01.940399 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8ae46a5ccab40f7807a83888b174f76" candidate_uuid: "749ce7bc297846778acfda0c2b755f2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9374a3b0ad334e898b753c0157b127ee" is_pre_election: true
I20260620 04:25:01.940730 1375 raft_consensus.cc:2454] T a8ae46a5ccab40f7807a83888b174f76 P efcf273e609247c8bf3b24dd56cf78e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 749ce7bc297846778acfda0c2b755f2e in term 0.
I20260620 04:25:01.940730 1107 raft_consensus.cc:2454] T a8ae46a5ccab40f7807a83888b174f76 P 9374a3b0ad334e898b753c0157b127ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 749ce7bc297846778acfda0c2b755f2e in term 0.
I20260620 04:25:01.941269 1175 leader_election.cc:302] T a8ae46a5ccab40f7807a83888b174f76 P 749ce7bc297846778acfda0c2b755f2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 749ce7bc297846778acfda0c2b755f2e, 9374a3b0ad334e898b753c0157b127ee, efcf273e609247c8bf3b24dd56cf78e5; no voters:
I20260620 04:25:01.941545 1447 raft_consensus.cc:2781] T a8ae46a5ccab40f7807a83888b174f76 P 749ce7bc297846778acfda0c2b755f2e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 04:25:01.941617 1447 raft_consensus.cc:482] T a8ae46a5ccab40f7807a83888b174f76 P 749ce7bc297846778acfda0c2b755f2e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 04:25:01.941648 1447 raft_consensus.cc:3037] T a8ae46a5ccab40f7807a83888b174f76 P 749ce7bc297846778acfda0c2b755f2e [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:25:02.022630 1447 raft_consensus.cc:504] T a8ae46a5ccab40f7807a83888b174f76 P 749ce7bc297846778acfda0c2b755f2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9374a3b0ad334e898b753c0157b127ee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35069 } } peers { permanent_uuid: "749ce7bc297846778acfda0c2b755f2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46225 } } peers { permanent_uuid: "efcf273e609247c8bf3b24dd56cf78e5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34219 } }
I20260620 04:25:02.023167 1447 leader_election.cc:288] T a8ae46a5ccab40f7807a83888b174f76 P 749ce7bc297846778acfda0c2b755f2e [CANDIDATE]: Term 1 election: Requested vote from peers 9374a3b0ad334e898b753c0157b127ee (127.0.98.1:35069), efcf273e609247c8bf3b24dd56cf78e5 (127.0.98.3:34219)
I20260620 04:25:02.023769 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8ae46a5ccab40f7807a83888b174f76" candidate_uuid: "749ce7bc297846778acfda0c2b755f2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efcf273e609247c8bf3b24dd56cf78e5"
I20260620 04:25:02.023787 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8ae46a5ccab40f7807a83888b174f76" candidate_uuid: "749ce7bc297846778acfda0c2b755f2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9374a3b0ad334e898b753c0157b127ee"
I20260620 04:25:02.023926 1107 raft_consensus.cc:3037] T a8ae46a5ccab40f7807a83888b174f76 P 9374a3b0ad334e898b753c0157b127ee [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:25:02.023917 1375 raft_consensus.cc:3037] T a8ae46a5ccab40f7807a83888b174f76 P efcf273e609247c8bf3b24dd56cf78e5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:25:02.025059 1375 raft_consensus.cc:2454] T a8ae46a5ccab40f7807a83888b174f76 P efcf273e609247c8bf3b24dd56cf78e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 749ce7bc297846778acfda0c2b755f2e in term 1.
I20260620 04:25:02.025197 1107 raft_consensus.cc:2454] T a8ae46a5ccab40f7807a83888b174f76 P 9374a3b0ad334e898b753c0157b127ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 749ce7bc297846778acfda0c2b755f2e in term 1.
I20260620 04:25:02.025461 1175 leader_election.cc:302] T a8ae46a5ccab40f7807a83888b174f76 P 749ce7bc297846778acfda0c2b755f2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 749ce7bc297846778acfda0c2b755f2e, efcf273e609247c8bf3b24dd56cf78e5; no voters:
I20260620 04:25:02.025641 1447 raft_consensus.cc:2781] T a8ae46a5ccab40f7807a83888b174f76 P 749ce7bc297846778acfda0c2b755f2e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:25:02.025861 1447 raft_consensus.cc:686] T a8ae46a5ccab40f7807a83888b174f76 P 749ce7bc297846778acfda0c2b755f2e [term 1 LEADER]: Becoming Leader. State: Replica: 749ce7bc297846778acfda0c2b755f2e, State: Running, Role: LEADER
I20260620 04:25:02.026258 1447 consensus_queue.cc:237] T a8ae46a5ccab40f7807a83888b174f76 P 749ce7bc297846778acfda0c2b755f2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9374a3b0ad334e898b753c0157b127ee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35069 } } peers { permanent_uuid: "749ce7bc297846778acfda0c2b755f2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46225 } } peers { permanent_uuid: "efcf273e609247c8bf3b24dd56cf78e5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34219 } }
I20260620 04:25:02.030438 958 catalog_manager.cc:5495] T a8ae46a5ccab40f7807a83888b174f76 P 749ce7bc297846778acfda0c2b755f2e reported cstate change: term changed from 0 to 1, leader changed from <none> to 749ce7bc297846778acfda0c2b755f2e (127.0.98.2). New cstate: current_term: 1 leader_uuid: "749ce7bc297846778acfda0c2b755f2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9374a3b0ad334e898b753c0157b127ee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35069 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "749ce7bc297846778acfda0c2b755f2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "efcf273e609247c8bf3b24dd56cf78e5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34219 } health_report { overall_health: UNKNOWN } } }
Jun 20 04:25:02 dist-test-slave-nk6d krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929499, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 04:25:02.086154 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:02.073876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36206 (local address 127.0.98.1:35069)
0620 04:25:02.074019 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:02.074024 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:02.074276 (+ 252us) server_negotiation.cc:407] Connection header received
0620 04:25:02.074400 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:02.074404 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:02.074455 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:02.074569 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:02.075457 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:02.076311 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:02.077535 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:02.077876 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:02.077982 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:02.082140 (+ 4158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:02.082158 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:02.082180 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:25:02.082217 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:02.084326 (+ 2109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:02.084771 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:02.084777 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:02.084778 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:02.084877 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:02.085176 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:02.085179 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:02.085181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:02.085472 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:02.085681 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:02.085866 (+ 185us) server_negotiation.cc:299] Negotiation successful
0620 04:25:02.085983 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
Jun 20 04:25:02 dist-test-slave-nk6d krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929502, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 04:25:02 dist-test-slave-nk6d krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929502, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 04:25:02.125715 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:02.117373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36220 (local address 127.0.98.1:35069)
0620 04:25:02.117527 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:02.117532 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:02.117666 (+ 134us) server_negotiation.cc:407] Connection header received
0620 04:25:02.117792 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:02.117795 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:02.117844 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:02.117937 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:02.118662 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:02.119239 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:02.120172 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:02.120419 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:02.120511 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:02.122924 (+ 2413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:02.122927 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:02.122930 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:25:02.122956 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:02.124134 (+ 1178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:02.124508 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:02.124510 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:02.124511 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:02.124578 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:02.124766 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:02.124768 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:02.124769 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:02.124983 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:02.125113 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:02.125446 (+ 333us) server_negotiation.cc:299] Negotiation successful
0620 04:25:02.125553 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
Jun 20 04:25:02 dist-test-slave-nk6d krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929502, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 04:25:02 dist-test-slave-nk6d krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929502, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 04:25:02.226718 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:02.217609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36230 (local address 127.0.98.1:35069)
0620 04:25:02.217736 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:02.217741 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:02.217882 (+ 141us) server_negotiation.cc:407] Connection header received
0620 04:25:02.218034 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:02.218036 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:02.218084 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:02.218156 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:02.219011 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:02.219638 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:02.220644 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:02.220872 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:02.220967 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:02.223647 (+ 2680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:02.223651 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:02.223653 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:25:02.223686 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:02.225204 (+ 1518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:02.225653 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:02.225656 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:02.225657 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:02.225767 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:02.225982 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:02.225984 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:02.225985 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:02.226157 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:02.226279 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:02.226444 (+ 165us) server_negotiation.cc:299] Negotiation successful
0620 04:25:02.226553 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20260620 04:25:02.228140 1067 tablet_service.cc:2908] Rejecting scan request for tablet a8ae46a5ccab40f7807a83888b174f76: Uninitialized: clean time has not yet been initialized
I20260620 04:25:02.230005 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 1024
I20260620 04:25:02.235455 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 1158
I20260620 04:25:02.240762 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 1292
I20260620 04:25:02.246052 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 930
2026-06-20T04:25:02Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (3895 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929496330398-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:25:03 dist-test-slave-nk6d krb5kdc[1465](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:25:03 dist-test-slave-nk6d krb5kdc[1465](info): set up 2 sockets
Jun 20 04:25:03 dist-test-slave-nk6d krb5kdc[1465](info): commencing operation
krb5kdc: starting...
W20260620 04:25:04.363513 392 mini_kdc.cc:121] Time spent starting KDC: real 2.096s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260620 04:25:04.506199 392 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.143s user 0.000s sys 0.002s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260620 04:25:04.669414 392 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.163s user 0.000s sys 0.002s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260620 04:25:04.965920 392 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.296s user 0.001s sys 0.001s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929496330398-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929496330398-392-0/krb5kdc/test-user.keytab.
W20260620 04:25:05.653324 392 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.687s user 0.001s sys 0.000s
Jun 20 04:25:05 dist-test-slave-nk6d krb5kdc[1465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929505, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:25:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:25:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 04:25:05.695665 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929496330398-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929496330398-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929496330398-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37763
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37763
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:25:05.802814 1478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:25:05.803093 1478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:25:05.803148 1478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:25:05.807114 1478 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:25:05.807175 1478 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 04:25:05.807197 1478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:25:05.807221 1478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:25:05.807242 1478 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:25:05.807266 1478 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:25:05.812467 1478 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929496330398-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929496330398-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37763
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37763
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1478
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929496330398-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:25:05.813465 1478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:25:05.814443 1478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:25:05.822517 1483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:25:05.822511 1486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:25:05.822734 1478 server_base.cc:1034] running on GCE node
W20260620 04:25:05.822916 1484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:25:05.823408 1478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:25:05.824502 1478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:25:05.825659 1478 hybrid_clock.cc:648] HybridClock initialized: now 1781929505825637 us; error 34 us; skew 500 ppm
Jun 20 04:25:05 dist-test-slave-nk6d krb5kdc[1465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929505, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:25:05.828294 1478 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 04:25:05.829161 1478 webserver.cc:460] Webserver started at http://127.0.98.62:41991/ using document root <none> and password file <none>
I20260620 04:25:05.829679 1478 fs_manager.cc:359] Metadata directory not provided
I20260620 04:25:05.829742 1478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:25:05.829931 1478 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:25:05.831882 1478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929496330398-392-0/minicluster-data/master-0/data/instance:
uuid: "fb5b3346cacb41ca9314bc02e26d17a7"
format_stamp: "Formatted at 2026-06-20 04:25:05 on dist-test-slave-nk6d"
server_key: "264db099619421a22ea959f2a9a56c8d"
server_key_iv: "9311af9961786696d841103c2382b68e"
server_key_version: "encryptionkey@0"
I20260620 04:25:05.832449 1478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929496330398-392-0/minicluster-data/master-0/wal/instance:
uuid: "fb5b3346cacb41ca9314bc02e26d17a7"
format_stamp: "Formatted at 2026-06-20 04:25:05 on dist-test-slave-nk6d"
server_key: "264db099619421a22ea959f2a9a56c8d"
server_key_iv: "9311af9961786696d841103c2382b68e"
server_key_version: "encryptionkey@0"
I20260620 04:25:05.836011 1478 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:25:05.838328 1493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:05.839551 1478 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:25:05.839668 1478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929496330398-392-0/minicluster-data/master-0/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929496330398-392-0/minicluster-data/master-0/wal
uuid: "fb5b3346cacb41ca9314bc02e26d17a7"
format_stamp: "Formatted at 2026-06-20 04:25:05 on dist-test-slave-nk6d"
server_key: "264db099619421a22ea959f2a9a56c8d"
server_key_iv: "9311af9961786696d841103c2382b68e"
server_key_version: "encryptionkey@0"
I20260620 04:25:05.839783 1478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929496330398-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929496330398-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929496330398-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:25:05.860038 1478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:25:05.863039 1478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:25:05.863206 1478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:25:05.870702 1478 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37763
I20260620 04:25:05.870729 1555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37763 every 8 connection(s)
I20260620 04:25:05.871582 1478 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:25:05.874218 1556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:25:05.879376 1556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb5b3346cacb41ca9314bc02e26d17a7: Bootstrap starting.
I20260620 04:25:05.881323 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 1478
I20260620 04:25:05.881431 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929496330398-392-0/minicluster-data/master-0/wal/instance
I20260620 04:25:05.881668 392 external_mini_cluster.cc:1428] Setting key 0c679ab34bbe0b88048373d8838f46a7
I20260620 04:25:05.881992 1556 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fb5b3346cacb41ca9314bc02e26d17a7: Neither blocks nor log segments found. Creating new log.
I20260620 04:25:05.882896 1556 log.cc:824] T 00000000000000000000000000000000 P fb5b3346cacb41ca9314bc02e26d17a7: Log is configured to *not* fsync() on all Append() calls
I20260620 04:25:05.884958 1556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb5b3346cacb41ca9314bc02e26d17a7: No bootstrap required, opened a new log
I20260620 04:25:05.889144 1556 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb5b3346cacb41ca9314bc02e26d17a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb5b3346cacb41ca9314bc02e26d17a7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37763 } }
I20260620 04:25:05.889375 1556 raft_consensus.cc:374] T 00000000000000000000000000000000 P fb5b3346cacb41ca9314bc02e26d17a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:25:05.889433 1556 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb5b3346cacb41ca9314bc02e26d17a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb5b3346cacb41ca9314bc02e26d17a7, State: Initialized, Role: FOLLOWER
I20260620 04:25:05.889940 1556 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb5b3346cacb41ca9314bc02e26d17a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb5b3346cacb41ca9314bc02e26d17a7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37763 } }
I20260620 04:25:05.890103 1556 raft_consensus.cc:388] T 00000000000000000000000000000000 P fb5b3346cacb41ca9314bc02e26d17a7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:25:05.890166 1556 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb5b3346cacb41ca9314bc02e26d17a7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:25:05.890256 1556 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb5b3346cacb41ca9314bc02e26d17a7 [term 0 FOLLOWER]: Advancing to term 1
Jun 20 04:25:05 dist-test-slave-nk6d krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929505, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:25:05.891172 1556 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb5b3346cacb41ca9314bc02e26d17a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb5b3346cacb41ca9314bc02e26d17a7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37763 } }
I20260620 04:25:05.891512 1556 leader_election.cc:302] T 00000000000000000000000000000000 P fb5b3346cacb41ca9314bc02e26d17a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb5b3346cacb41ca9314bc02e26d17a7; no voters:
I20260620 04:25:05.891805 1556 leader_election.cc:288] T 00000000000000000000000000000000 P fb5b3346cacb41ca9314bc02e26d17a7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:25:05.891903 1561 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb5b3346cacb41ca9314bc02e26d17a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:25:05.892144 1561 raft_consensus.cc:686] T 00000000000000000000000000000000 P fb5b3346cacb41ca9314bc02e26d17a7 [term 1 LEADER]: Becoming Leader. State: Replica: fb5b3346cacb41ca9314bc02e26d17a7, State: Running, Role: LEADER
I20260620 04:25:05.892501 1561 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb5b3346cacb41ca9314bc02e26d17a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb5b3346cacb41ca9314bc02e26d17a7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37763 } }
I20260620 04:25:05.893033 1556 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb5b3346cacb41ca9314bc02e26d17a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:25:05.893780 1561 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb5b3346cacb41ca9314bc02e26d17a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb5b3346cacb41ca9314bc02e26d17a7. Latest consensus state: current_term: 1 leader_uuid: "fb5b3346cacb41ca9314bc02e26d17a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb5b3346cacb41ca9314bc02e26d17a7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37763 } } }
I20260620 04:25:05.894492 1561 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb5b3346cacb41ca9314bc02e26d17a7 [sys.catalog]: This master's current role is: LEADER
I20260620 04:25:05.894383 1562 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb5b3346cacb41ca9314bc02e26d17a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fb5b3346cacb41ca9314bc02e26d17a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb5b3346cacb41ca9314bc02e26d17a7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37763 } } }
I20260620 04:25:05.894797 1562 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb5b3346cacb41ca9314bc02e26d17a7 [sys.catalog]: This master's current role is: LEADER
I20260620 04:25:05.897105 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:05.882779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50730 (local address 127.0.98.62:37763)
0620 04:25:05.883195 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:05.883210 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:05.883250 (+ 40us) server_negotiation.cc:407] Connection header received
0620 04:25:05.884042 (+ 792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:05.884063 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:05.884398 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:05.884671 (+ 273us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:05.885510 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:05.886890 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:05.888091 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:05.888665 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:05.888795 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:05.891454 (+ 2659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:05.891459 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:05.891472 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:25:05.891497 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:05.893533 (+ 2036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:05.894011 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:05.894017 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:05.894022 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:05.894161 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:05.894493 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:05.894496 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:05.894497 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:05.895000 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:05.895196 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:05.895894 (+ 698us) server_negotiation.cc:299] Negotiation successful
0620 04:25:05.896187 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":98,"threads_started":1}
W20260620 04:25:05.897594 1576 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fb5b3346cacb41ca9314bc02e26d17a7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 04:25:05.897655 1576 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 04:25:05.897715 1575 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:25:05.898488 1575 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:25:05.903496 1575 catalog_manager.cc:1347] Generated new cluster ID: c0b189b5c3154405ad69e74c0e7c6844
I20260620 04:25:05.903543 1575 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:25:05.918571 1575 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:25:05.919486 1575 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:25:05.926249 1575 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fb5b3346cacb41ca9314bc02e26d17a7: Generated new TSK 0
I20260620 04:25:05.926964 1575 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929496330398-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929496330398-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 04:25:05.958650 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929496330398-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929496330398-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929496330398-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929496330398-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37763
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929496330398-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:25:06.068212 1582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:25:06.068490 1582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:25:06.068535 1582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:25:06.072450 1582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:25:06.072510 1582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:25:06.072548 1582 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 04:25:06.072580 1582 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 04:25:06.077831 1582 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929496330398-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929496330398-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929496330398-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929496330398-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:37763
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1582
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929496330398-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:25:06.078846 1582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:25:06.079821 1582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:25:06.089296 1587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:25:06.089298 1590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:25:06.089289 1588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:25:06.089643 1582 server_base.cc:1034] running on GCE node
I20260620 04:25:06.090153 1582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:25:06.090771 1582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:25:06.091925 1582 hybrid_clock.cc:648] HybridClock initialized: now 1781929506091904 us; error 37 us; skew 500 ppm
Jun 20 04:25:06 dist-test-slave-nk6d krb5kdc[1465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929506, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:25:06.094699 1582 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 04:25:06.095500 1582 webserver.cc:460] Webserver started at http://127.0.98.1:35289/ using document root <none> and password file <none>
I20260620 04:25:06.096047 1582 fs_manager.cc:359] Metadata directory not provided
I20260620 04:25:06.096115 1582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:25:06.096307 1582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:25:06.098309 1582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929496330398-392-0/minicluster-data/ts-0/data/instance:
uuid: "92efaefa517c473cadbf761e21859ff4"
format_stamp: "Formatted at 2026-06-20 04:25:06 on dist-test-slave-nk6d"
server_key: "59bc7cfa94b2aeb05c1c23739621e3f8"
server_key_iv: "4f17413cfd7e071a973548c73d7aaa39"
server_key_version: "encryptionkey@0"
I20260620 04:25:06.098834 1582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929496330398-392-0/minicluster-data/ts-0/wal/instance:
uuid: "92efaefa517c473cadbf761e21859ff4"
format_stamp: "Formatted at 2026-06-20 04:25:06 on dist-test-slave-nk6d"
server_key: "59bc7cfa94b2aeb05c1c23739621e3f8"
server_key_iv: "4f17413cfd7e071a973548c73d7aaa39"
server_key_version: "encryptionkey@0"
I20260620 04:25:06.102474 1582 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 04:25:06.104959 1597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:06.106212 1582 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:25:06.106340 1582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929496330398-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929496330398-392-0/minicluster-data/ts-0/wal
uuid: "92efaefa517c473cadbf761e21859ff4"
format_stamp: "Formatted at 2026-06-20 04:25:06 on dist-test-slave-nk6d"
server_key: "59bc7cfa94b2aeb05c1c23739621e3f8"
server_key_iv: "4f17413cfd7e071a973548c73d7aaa39"
server_key_version: "encryptionkey@0"
I20260620 04:25:06.106447 1582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929496330398-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929496330398-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929496330398-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:25:06.117370 1582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:25:06.120342 1582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:25:06.120518 1582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:25:06.121891 1582 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:25:06.121938 1582 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:06.121989 1582 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:25:06.122018 1582 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 04:25:06 dist-test-slave-nk6d krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929506, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:25:06.131798 1582 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45753
I20260620 04:25:06.131866 1712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45753 every 8 connection(s)
I20260620 04:25:06.132798 1582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929496330398-392-0/minicluster-data/ts-0/data/info.pb
I20260620 04:25:06.134500 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 1582
I20260620 04:25:06.134604 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929496330398-392-0/minicluster-data/ts-0/wal/instance
I20260620 04:25:06.134790 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:06.122800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33565 (local address 127.0.98.62:37763)
0620 04:25:06.122939 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:06.122944 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:06.123789 (+ 845us) server_negotiation.cc:407] Connection header received
0620 04:25:06.124933 (+ 1144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:06.124936 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:06.124995 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:06.125067 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:06.126293 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:06.126812 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:06.128047 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:06.128282 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:06.128373 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:06.131000 (+ 2627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:06.131002 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:06.131004 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:25:06.131030 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:06.132678 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:06.133256 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:06.133260 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:06.133262 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:06.133352 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:06.133717 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:06.133720 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:06.133722 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:06.133911 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:06.133994 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:06.134495 (+ 501us) server_negotiation.cc:299] Negotiation successful
0620 04:25:06.134629 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 04:25:06.134877 392 external_mini_cluster.cc:1428] Setting key 739656d0be98849a76360959bc0bc9d2
I20260620 04:25:06.135598 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:06.123178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37763 (local address 127.0.98.1:33565)
0620 04:25:06.123623 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:06.123671 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:06.124724 (+ 1053us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:06.125247 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:06.125254 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:06.125587 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:06.126150 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:06.126164 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:06.126933 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:06.126937 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:06.127919 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:06.127929 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:06.128437 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:06.128441 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:06.128597 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:06.129200 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:06.129216 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:06.130826 (+ 1610us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:25:06.132814 (+ 1988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:06.132820 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:06.132835 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:06.133148 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:06.133463 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:06.133466 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:06.133468 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:06.133626 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:06.134007 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:25:06.134014 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:25:06.134244 (+ 230us) client_negotiation.cc:769] Sending connection context
0620 04:25:06.134436 (+ 192us) client_negotiation.cc:240] Negotiation successful
0620 04:25:06.134739 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":110,"threads_started":1}
I20260620 04:25:06.136773 1713 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37763
I20260620 04:25:06.136916 1713 heartbeater.cc:464] Registering TS with master...
I20260620 04:25:06.137393 1713 heartbeater.cc:511] Master 127.0.98.62:37763 requested a full tablet report, sending...
I20260620 04:25:06.138931 1509 ts_manager.cc:194] Registered new tserver with Master: 92efaefa517c473cadbf761e21859ff4 (127.0.98.1:45753)
I20260620 04:25:06.140439 1509 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:33565
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 04:25:06.148644 1720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:06.138929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37763 (local address 127.0.98.1:46527)
0620 04:25:06.141140 (+ 2211us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:06.141154 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:06.141279 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:06.141541 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:06.141544 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:06.141773 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:06.141908 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:06.141916 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:06.142933 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:06.142937 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:06.143865 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:06.143877 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:06.144365 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:06.144368 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:06.144467 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:06.144934 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:06.144948 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:06.145351 (+ 403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:25:06.147069 (+ 1718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:06.147072 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:06.147074 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:06.147248 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:06.147566 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:06.147569 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:06.147571 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:06.147655 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:06.148067 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:25:06.148070 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:25:06.148163 (+ 93us) client_negotiation.cc:769] Sending connection context
0620 04:25:06.148264 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 04:25:06.148459 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2129,"thread_start_us":100,"threads_started":1}
I20260620 04:25:06.148669 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:06.139065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46527 (local address 127.0.98.62:37763)
0620 04:25:06.139175 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:06.139179 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:06.141217 (+ 2038us) server_negotiation.cc:407] Connection header received
0620 04:25:06.141378 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:06.141382 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:06.141429 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:06.141512 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:06.142025 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:06.142780 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:06.144013 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:06.144228 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:06.144325 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:06.145508 (+ 1183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:06.145511 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:06.145513 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:25:06.145535 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:06.146976 (+ 1441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:06.147354 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:06.147357 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:06.147359 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:06.147437 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:06.147738 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:06.147741 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:06.147743 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:06.147932 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:06.148077 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:06.148345 (+ 268us) server_negotiation.cc:299] Negotiation successful
0620 04:25:06.148516 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929496330398-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929496330398-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 04:25:06.160988 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929496330398-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929496330398-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929496330398-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929496330398-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37763
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929496330398-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:25:06.264815 1722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:25:06.265072 1722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:25:06.265115 1722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:25:06.269037 1722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:25:06.269090 1722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:25:06.269129 1722 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 04:25:06.269158 1722 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 04:25:06.274341 1722 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929496330398-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929496330398-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929496330398-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929496330398-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:37763
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1722
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929496330398-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:25:06.275310 1722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:25:06.276263 1722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:25:06.285360 1728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:25:06.285385 1730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:25:06.285673 1722 server_base.cc:1034] running on GCE node
W20260620 04:25:06.285392 1727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:25:06.286268 1722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:25:06.286873 1722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:25:06.288044 1722 hybrid_clock.cc:648] HybridClock initialized: now 1781929506288016 us; error 39 us; skew 500 ppm
Jun 20 04:25:06 dist-test-slave-nk6d krb5kdc[1465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929506, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:25:06.290547 1722 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 04:25:06.291364 1722 webserver.cc:460] Webserver started at http://127.0.98.2:35583/ using document root <none> and password file <none>
I20260620 04:25:06.291849 1722 fs_manager.cc:359] Metadata directory not provided
I20260620 04:25:06.291903 1722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:25:06.292105 1722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:25:06.294113 1722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929496330398-392-0/minicluster-data/ts-1/data/instance:
uuid: "8af8ecc3c97a4bc9908aa7f03122832c"
format_stamp: "Formatted at 2026-06-20 04:25:06 on dist-test-slave-nk6d"
server_key: "386ba0f3813c7269d21a5b7c7b94a7f0"
server_key_iv: "77c5554362adc335ff9da1193412856f"
server_key_version: "encryptionkey@0"
I20260620 04:25:06.294591 1722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929496330398-392-0/minicluster-data/ts-1/wal/instance:
uuid: "8af8ecc3c97a4bc9908aa7f03122832c"
format_stamp: "Formatted at 2026-06-20 04:25:06 on dist-test-slave-nk6d"
server_key: "386ba0f3813c7269d21a5b7c7b94a7f0"
server_key_iv: "77c5554362adc335ff9da1193412856f"
server_key_version: "encryptionkey@0"
I20260620 04:25:06.298239 1722 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:25:06.300526 1737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:06.301651 1722 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:25:06.301740 1722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929496330398-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929496330398-392-0/minicluster-data/ts-1/wal
uuid: "8af8ecc3c97a4bc9908aa7f03122832c"
format_stamp: "Formatted at 2026-06-20 04:25:06 on dist-test-slave-nk6d"
server_key: "386ba0f3813c7269d21a5b7c7b94a7f0"
server_key_iv: "77c5554362adc335ff9da1193412856f"
server_key_version: "encryptionkey@0"
I20260620 04:25:06.301836 1722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929496330398-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929496330398-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929496330398-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:25:06.318113 1722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:25:06.320694 1722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:25:06.320858 1722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:25:06.322202 1722 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:25:06.322248 1722 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:06.322285 1722 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:25:06.322309 1722 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 04:25:06 dist-test-slave-nk6d krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929506, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:25:06.331905 1722 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:32803
I20260620 04:25:06.332296 1852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:32803 every 8 connection(s)
I20260620 04:25:06.332890 1722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929496330398-392-0/minicluster-data/ts-1/data/info.pb
I20260620 04:25:06.334756 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:06.322923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42717 (local address 127.0.98.62:37763)
0620 04:25:06.323062 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:06.323067 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:06.323916 (+ 849us) server_negotiation.cc:407] Connection header received
0620 04:25:06.324927 (+ 1011us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:06.324930 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:06.324995 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:06.325068 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:06.326297 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:06.326814 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:06.327940 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:06.328314 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:06.328422 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:06.331341 (+ 2919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:06.331345 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:06.331348 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:25:06.331378 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:06.332898 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:06.333468 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:06.333484 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:06.333486 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:06.333576 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:06.333866 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:06.333869 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:06.333870 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:06.334043 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:06.334118 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:06.334546 (+ 428us) server_negotiation.cc:299] Negotiation successful
0620 04:25:06.334640 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 04:25:06.335547 1746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:06.323318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37763 (local address 127.0.98.2:42717)
0620 04:25:06.323753 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:06.323798 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:06.324707 (+ 909us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:06.325239 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:06.325245 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:06.325562 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:06.326153 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:06.326174 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:06.326926 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:06.326930 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:06.327823 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:06.327833 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:06.328487 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:06.328493 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:06.328693 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:06.329328 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:06.329346 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:06.331172 (+ 1826us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:25:06.333014 (+ 1842us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:06.333019 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:06.333033 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:06.333349 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:06.333656 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:06.333658 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:06.333660 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:06.333783 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:06.334119 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:25:06.334124 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:25:06.334335 (+ 211us) client_negotiation.cc:769] Sending connection context
0620 04:25:06.334513 (+ 178us) client_negotiation.cc:240] Negotiation successful
0620 04:25:06.334750 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":119,"threads_started":1}
I20260620 04:25:06.336236 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 1722
I20260620 04:25:06.336347 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929496330398-392-0/minicluster-data/ts-1/wal/instance
I20260620 04:25:06.336604 392 external_mini_cluster.cc:1428] Setting key 12418ad9ab165843f830715651be8dda
I20260620 04:25:06.337710 1853 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37763
I20260620 04:25:06.339605 1853 heartbeater.cc:464] Registering TS with master...
I20260620 04:25:06.340097 1853 heartbeater.cc:511] Master 127.0.98.62:37763 requested a full tablet report, sending...
I20260620 04:25:06.341010 1510 ts_manager.cc:194] Registered new tserver with Master: 8af8ecc3c97a4bc9908aa7f03122832c (127.0.98.2:32803)
I20260620 04:25:06.341696 1510 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42717
I20260620 04:25:06.345511 1859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:06.338895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37763 (local address 127.0.98.2:59193)
0620 04:25:06.339155 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:06.339170 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:06.339263 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:06.339485 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:06.339488 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:06.339705 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:06.339822 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:06.339827 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:06.340635 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:06.340638 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:06.341310 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:06.341319 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:06.341715 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:06.341718 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:06.341801 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:06.342275 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:06.342286 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:06.342587 (+ 301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:25:06.344094 (+ 1507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:06.344097 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:06.344100 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:06.344344 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:06.344586 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:06.344589 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:06.344591 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:06.344667 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:06.345040 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:25:06.345043 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:25:06.345121 (+ 78us) client_negotiation.cc:769] Sending connection context
0620 04:25:06.345202 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 04:25:06.345324 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":92,"threads_started":1}
I20260620 04:25:06.345520 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:06.338996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59193 (local address 127.0.98.62:37763)
0620 04:25:06.339176 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:06.339179 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:06.339231 (+ 52us) server_negotiation.cc:407] Connection header received
0620 04:25:06.339329 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:06.339331 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:06.339364 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:06.339432 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:06.339919 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:06.340519 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:06.341410 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:06.341625 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:06.341705 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:06.342704 (+ 999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:06.342708 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:06.342711 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:25:06.342733 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:06.344000 (+ 1267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:06.344419 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:06.344422 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:06.344424 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:06.344511 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:06.344738 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:06.344741 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:06.344743 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:06.344935 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:06.345017 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:06.345258 (+ 241us) server_negotiation.cc:299] Negotiation successful
0620 04:25:06.345351 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929496330398-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929496330398-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 04:25:06.361155 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929496330398-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929496330398-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929496330398-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929496330398-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37763
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929496330398-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:25:06.467703 1862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:25:06.467979 1862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:25:06.468042 1862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:25:06.471925 1862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:25:06.471997 1862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:25:06.472035 1862 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 04:25:06.472075 1862 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 04:25:06.477154 1862 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929496330398-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929496330398-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929496330398-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929496330398-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:37763
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1862
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929496330398-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:25:06.478158 1862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:25:06.479137 1862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:25:06.488236 1867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:25:06.488319 1870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:25:06.488281 1868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:25:06.488444 1862 server_base.cc:1034] running on GCE node
I20260620 04:25:06.488965 1862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:25:06.489620 1862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:25:06.490770 1862 hybrid_clock.cc:648] HybridClock initialized: now 1781929506490760 us; error 52 us; skew 500 ppm
Jun 20 04:25:06 dist-test-slave-nk6d krb5kdc[1465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929506, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:25:06.493459 1862 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 04:25:06.494340 1862 webserver.cc:460] Webserver started at http://127.0.98.3:44815/ using document root <none> and password file <none>
I20260620 04:25:06.494854 1862 fs_manager.cc:359] Metadata directory not provided
I20260620 04:25:06.494911 1862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:25:06.495088 1862 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:25:06.497083 1862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929496330398-392-0/minicluster-data/ts-2/data/instance:
uuid: "fc94a650d15247b7b08bd5c3eeaa41e6"
format_stamp: "Formatted at 2026-06-20 04:25:06 on dist-test-slave-nk6d"
server_key: "5961432a2f2c6f2a8f90166bcffb8414"
server_key_iv: "3618666c25693a9ad3c0b079dd777507"
server_key_version: "encryptionkey@0"
I20260620 04:25:06.497586 1862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929496330398-392-0/minicluster-data/ts-2/wal/instance:
uuid: "fc94a650d15247b7b08bd5c3eeaa41e6"
format_stamp: "Formatted at 2026-06-20 04:25:06 on dist-test-slave-nk6d"
server_key: "5961432a2f2c6f2a8f90166bcffb8414"
server_key_iv: "3618666c25693a9ad3c0b079dd777507"
server_key_version: "encryptionkey@0"
I20260620 04:25:06.501102 1862 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260620 04:25:06.503357 1877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:06.504591 1862 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:25:06.504688 1862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929496330398-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929496330398-392-0/minicluster-data/ts-2/wal
uuid: "fc94a650d15247b7b08bd5c3eeaa41e6"
format_stamp: "Formatted at 2026-06-20 04:25:06 on dist-test-slave-nk6d"
server_key: "5961432a2f2c6f2a8f90166bcffb8414"
server_key_iv: "3618666c25693a9ad3c0b079dd777507"
server_key_version: "encryptionkey@0"
I20260620 04:25:06.504776 1862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929496330398-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929496330398-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929496330398-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:25:06.519090 1862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:25:06.522123 1862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:25:06.522294 1862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:25:06.523696 1862 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:25:06.523744 1862 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:06.523797 1862 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:25:06.523823 1862 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:06.533445 1862 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37745
Jun 20 04:25:06 dist-test-slave-nk6d krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929506, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:25:06.533596 1992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37745 every 8 connection(s)
I20260620 04:25:06.534313 1862 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929496330398-392-0/minicluster-data/ts-2/data/info.pb
I20260620 04:25:06.536487 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 1862
I20260620 04:25:06.536587 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929496330398-392-0/minicluster-data/ts-2/wal/instance
I20260620 04:25:06.536827 392 external_mini_cluster.cc:1428] Setting key 734b690005064500a5ba3c41e5d1ae3e
I20260620 04:25:06.538103 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:06.524532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37237 (local address 127.0.98.62:37763)
0620 04:25:06.524659 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:06.524663 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:06.525635 (+ 972us) server_negotiation.cc:407] Connection header received
0620 04:25:06.526778 (+ 1143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:06.526782 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:06.526868 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:06.526966 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:06.528496 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:06.529295 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:06.530970 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:06.531193 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:06.531278 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:06.534200 (+ 2922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:06.534205 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:06.534208 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:25:06.534239 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:06.535723 (+ 1484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:06.536336 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:06.536340 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:06.536343 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:06.536426 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:06.536871 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:06.536875 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:06.536877 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:06.537102 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:06.537202 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:06.537812 (+ 610us) server_negotiation.cc:299] Negotiation successful
0620 04:25:06.537929 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 04:25:06.538995 1886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:06.524967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37763 (local address 127.0.98.3:37237)
0620 04:25:06.525479 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:06.525522 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:06.526515 (+ 993us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:06.527183 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:06.527192 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:06.527633 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:06.528343 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:06.528364 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:06.529431 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:06.529435 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:06.530835 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:06.530845 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:06.531334 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:06.531339 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:06.531550 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:06.532253 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:06.532276 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:06.534024 (+ 1748us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:25:06.535845 (+ 1821us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:06.535851 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:06.535866 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:06.536200 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:06.536563 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:06.536567 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:06.536569 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:06.536733 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:06.537243 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:25:06.537250 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:25:06.537530 (+ 280us) client_negotiation.cc:769] Sending connection context
0620 04:25:06.537734 (+ 204us) client_negotiation.cc:240] Negotiation successful
0620 04:25:06.538064 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":127,"threads_started":1}
I20260620 04:25:06.540045 1993 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37763
I20260620 04:25:06.540175 1993 heartbeater.cc:464] Registering TS with master...
I20260620 04:25:06.540731 1993 heartbeater.cc:511] Master 127.0.98.62:37763 requested a full tablet report, sending...
I20260620 04:25:06.541729 1509 ts_manager.cc:194] Registered new tserver with Master: fc94a650d15247b7b08bd5c3eeaa41e6 (127.0.98.3:37745)
I20260620 04:25:06.542443 1509 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:37237
I20260620 04:25:06.549129 1999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:06.542260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37763 (local address 127.0.98.3:33955)
0620 04:25:06.542506 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:06.542519 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:06.542645 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:06.543107 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:06.543109 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:06.543263 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:06.543350 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:06.543355 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:06.544334 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:06.544337 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:06.544872 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:06.544879 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:06.545245 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:06.545247 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:06.545309 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:06.545755 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:06.545767 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:06.546064 (+ 297us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:25:06.547609 (+ 1545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:06.547611 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:06.547613 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:06.547830 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:06.548199 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:06.548202 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:06.548203 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:06.548287 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:06.548684 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:25:06.548687 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:25:06.548772 (+ 85us) client_negotiation.cc:769] Sending connection context
0620 04:25:06.548858 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 04:25:06.548989 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":102,"threads_started":1}
I20260620 04:25:06.549212 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:06.542339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33955 (local address 127.0.98.62:37763)
0620 04:25:06.542433 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:06.542437 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:06.542587 (+ 150us) server_negotiation.cc:407] Connection header received
0620 04:25:06.542955 (+ 368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:06.542958 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:06.543012 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:06.543091 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:06.543481 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:06.544228 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:06.544960 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:06.545167 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:06.545238 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:06.546173 (+ 935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:06.546175 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:06.546178 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:25:06.546196 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:06.547493 (+ 1297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:06.547996 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:06.548000 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:06.548002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:06.548088 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:06.548404 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:06.548410 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:06.548412 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:06.548592 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:06.548675 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:06.548950 (+ 275us) server_negotiation.cc:299] Negotiation successful
0620 04:25:06.549074 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 04:25:06.550724 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 04:25:06 dist-test-slave-nk6d krb5kdc[1465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929506, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 04:25:06 dist-test-slave-nk6d krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929506, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:25:06.574262 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:06.566800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50734 (local address 127.0.98.62:37763)
0620 04:25:06.566924 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:06.566927 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:06.566942 (+ 15us) server_negotiation.cc:407] Connection header received
0620 04:25:06.566984 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:06.566986 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:06.567030 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:06.567109 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:06.567715 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:06.568339 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:06.569186 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:06.569401 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:06.569486 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:06.571641 (+ 2155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:06.571644 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:06.571646 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:25:06.571668 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:06.572980 (+ 1312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:06.573346 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:06.573349 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:06.573350 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:06.573415 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:06.573621 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:06.573622 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:06.573623 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:06.573774 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:06.573866 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:06.574057 (+ 191us) server_negotiation.cc:299] Negotiation successful
0620 04:25:06.574145 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 04:25:06.576822 1510 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50734:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 04:25:06.579653 1510 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 04:25:06.593004 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:06.587398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37745 (local address 127.0.0.1:51314)
0620 04:25:06.588410 (+ 1012us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:06.588422 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:06.588514 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:06.589049 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:06.589055 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:06.589077 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:25:06.589312 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:06.589322 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:06.590330 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:06.590333 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:06.591540 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:06.591549 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:06.592487 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:06.592491 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:06.592600 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:06.592638 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 04:25:06.592741 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 04:25:06.592858 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":955,"mutex_wait_us":31,"thread_start_us":102,"threads_started":1}
I20260620 04:25:06.593024 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:06.587572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51314 (local address 127.0.98.3:37745)
0620 04:25:06.588057 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:06.588069 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:06.588610 (+ 541us) server_negotiation.cc:407] Connection header received
0620 04:25:06.588691 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:06.588696 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:06.588899 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:06.589058 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:25:06.589417 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:06.590227 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:06.591678 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:06.592382 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:06.592475 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:06.592644 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:06.592776 (+ 132us) server_negotiation.cc:299] Negotiation successful
0620 04:25:06.592868 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":93,"threads_started":1}
I20260620 04:25:06.593753 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:06.588138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45753 (local address 127.0.0.1:43140)
0620 04:25:06.588436 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:06.588448 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:06.588529 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:06.589209 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:06.589212 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:06.589227 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:25:06.589346 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:06.589352 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:06.590783 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:06.590786 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:06.591949 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:06.591982 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:06.593222 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:06.593226 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:06.593363 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:06.593436 (+ 73us) client_negotiation.cc:769] Sending connection context
0620 04:25:06.593505 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 04:25:06.593620 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"mutex_wait_us":143,"spinlock_wait_cycles":28416,"thread_start_us":94,"threads_started":1}
I20260620 04:25:06.593768 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:06.588312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43140 (local address 127.0.98.1:45753)
0620 04:25:06.588591 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:06.588601 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:06.588623 (+ 22us) server_negotiation.cc:407] Connection header received
0620 04:25:06.588704 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:06.588709 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:06.588911 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:06.589062 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:25:06.589467 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:06.590668 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:06.592068 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:06.592833 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:06.592991 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:06.593160 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:06.593554 (+ 394us) server_negotiation.cc:299] Negotiation successful
0620 04:25:06.593647 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":97,"threads_started":1}
I20260620 04:25:06.593856 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:06.588110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49882 (local address 127.0.98.2:32803)
0620 04:25:06.588748 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:06.588758 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:06.588776 (+ 18us) server_negotiation.cc:407] Connection header received
0620 04:25:06.588835 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:06.588839 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:06.588977 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:06.589253 (+ 276us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:25:06.589650 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:06.590748 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:06.592020 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:06.592805 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:06.593063 (+ 258us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:06.593411 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:06.593619 (+ 208us) server_negotiation.cc:299] Negotiation successful
0620 04:25:06.593727 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":108,"threads_started":1}
I20260620 04:25:06.596192 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:06.587287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32803 (local address 127.0.0.1:49882)
0620 04:25:06.587690 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:06.587735 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:06.587845 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:06.589111 (+ 1266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:06.589114 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:06.589130 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:25:06.589345 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:06.589354 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:06.590846 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:06.590849 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:06.591891 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:06.591900 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:06.593222 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:06.593226 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:06.593401 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:06.593443 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 04:25:06.593514 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 04:25:06.596067 (+ 2553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"mutex_wait_us":2587,"thread_start_us":123,"threads_started":1}
I20260620 04:25:06.596179 1647 tablet_service.cc:1467] Processing CreateTablet for tablet 7530c61db844470b808963ae5e963ac0 (DEFAULT_TABLE table=test-table [id=0e866f4cc59141bea5b118a986488e85]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:25:06.596169 1927 tablet_service.cc:1467] Processing CreateTablet for tablet 7530c61db844470b808963ae5e963ac0 (DEFAULT_TABLE table=test-table [id=0e866f4cc59141bea5b118a986488e85]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:25:06.597167 1647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7530c61db844470b808963ae5e963ac0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:25:06.597167 1927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7530c61db844470b808963ae5e963ac0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:25:06.599066 1787 tablet_service.cc:1467] Processing CreateTablet for tablet 7530c61db844470b808963ae5e963ac0 (DEFAULT_TABLE table=test-table [id=0e866f4cc59141bea5b118a986488e85]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:25:06.600085 1787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7530c61db844470b808963ae5e963ac0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:25:06.603154 2016 tablet_bootstrap.cc:492] T 7530c61db844470b808963ae5e963ac0 P 92efaefa517c473cadbf761e21859ff4: Bootstrap starting.
I20260620 04:25:06.605170 2017 tablet_bootstrap.cc:492] T 7530c61db844470b808963ae5e963ac0 P fc94a650d15247b7b08bd5c3eeaa41e6: Bootstrap starting.
I20260620 04:25:06.605401 2016 tablet_bootstrap.cc:654] T 7530c61db844470b808963ae5e963ac0 P 92efaefa517c473cadbf761e21859ff4: Neither blocks nor log segments found. Creating new log.
I20260620 04:25:06.606150 2016 log.cc:824] T 7530c61db844470b808963ae5e963ac0 P 92efaefa517c473cadbf761e21859ff4: Log is configured to *not* fsync() on all Append() calls
I20260620 04:25:06.607255 2017 tablet_bootstrap.cc:654] T 7530c61db844470b808963ae5e963ac0 P fc94a650d15247b7b08bd5c3eeaa41e6: Neither blocks nor log segments found. Creating new log.
I20260620 04:25:06.607780 2016 tablet_bootstrap.cc:492] T 7530c61db844470b808963ae5e963ac0 P 92efaefa517c473cadbf761e21859ff4: No bootstrap required, opened a new log
I20260620 04:25:06.608021 2016 ts_tablet_manager.cc:1397] T 7530c61db844470b808963ae5e963ac0 P 92efaefa517c473cadbf761e21859ff4: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260620 04:25:06.608141 2017 log.cc:824] T 7530c61db844470b808963ae5e963ac0 P fc94a650d15247b7b08bd5c3eeaa41e6: Log is configured to *not* fsync() on all Append() calls
I20260620 04:25:06.609864 2017 tablet_bootstrap.cc:492] T 7530c61db844470b808963ae5e963ac0 P fc94a650d15247b7b08bd5c3eeaa41e6: No bootstrap required, opened a new log
I20260620 04:25:06.610011 2017 ts_tablet_manager.cc:1397] T 7530c61db844470b808963ae5e963ac0 P fc94a650d15247b7b08bd5c3eeaa41e6: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260620 04:25:06.611469 2016 raft_consensus.cc:348] T 7530c61db844470b808963ae5e963ac0 P 92efaefa517c473cadbf761e21859ff4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8af8ecc3c97a4bc9908aa7f03122832c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32803 } } peers { permanent_uuid: "fc94a650d15247b7b08bd5c3eeaa41e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37745 } } peers { permanent_uuid: "92efaefa517c473cadbf761e21859ff4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45753 } }
I20260620 04:25:06.611673 2016 raft_consensus.cc:374] T 7530c61db844470b808963ae5e963ac0 P 92efaefa517c473cadbf761e21859ff4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:25:06.611726 2016 raft_consensus.cc:729] T 7530c61db844470b808963ae5e963ac0 P 92efaefa517c473cadbf761e21859ff4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92efaefa517c473cadbf761e21859ff4, State: Initialized, Role: FOLLOWER
I20260620 04:25:06.612141 2016 consensus_queue.cc:260] T 7530c61db844470b808963ae5e963ac0 P 92efaefa517c473cadbf761e21859ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8af8ecc3c97a4bc9908aa7f03122832c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32803 } } peers { permanent_uuid: "fc94a650d15247b7b08bd5c3eeaa41e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37745 } } peers { permanent_uuid: "92efaefa517c473cadbf761e21859ff4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45753 } }
I20260620 04:25:06.612721 1713 heartbeater.cc:503] Master 127.0.98.62:37763 was elected leader, sending a full tablet report...
I20260620 04:25:06.612833 2017 raft_consensus.cc:348] T 7530c61db844470b808963ae5e963ac0 P fc94a650d15247b7b08bd5c3eeaa41e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8af8ecc3c97a4bc9908aa7f03122832c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32803 } } peers { permanent_uuid: "fc94a650d15247b7b08bd5c3eeaa41e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37745 } } peers { permanent_uuid: "92efaefa517c473cadbf761e21859ff4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45753 } }
I20260620 04:25:06.613037 2017 raft_consensus.cc:374] T 7530c61db844470b808963ae5e963ac0 P fc94a650d15247b7b08bd5c3eeaa41e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:25:06.613068 2017 raft_consensus.cc:729] T 7530c61db844470b808963ae5e963ac0 P fc94a650d15247b7b08bd5c3eeaa41e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc94a650d15247b7b08bd5c3eeaa41e6, State: Initialized, Role: FOLLOWER
I20260620 04:25:06.613070 2016 ts_tablet_manager.cc:1428] T 7530c61db844470b808963ae5e963ac0 P 92efaefa517c473cadbf761e21859ff4: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260620 04:25:06.613436 2019 tablet_bootstrap.cc:492] T 7530c61db844470b808963ae5e963ac0 P 8af8ecc3c97a4bc9908aa7f03122832c: Bootstrap starting.
I20260620 04:25:06.613416 2017 consensus_queue.cc:260] T 7530c61db844470b808963ae5e963ac0 P fc94a650d15247b7b08bd5c3eeaa41e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8af8ecc3c97a4bc9908aa7f03122832c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32803 } } peers { permanent_uuid: "fc94a650d15247b7b08bd5c3eeaa41e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37745 } } peers { permanent_uuid: "92efaefa517c473cadbf761e21859ff4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45753 } }
I20260620 04:25:06.614228 1993 heartbeater.cc:503] Master 127.0.98.62:37763 was elected leader, sending a full tablet report...
I20260620 04:25:06.614279 2017 ts_tablet_manager.cc:1428] T 7530c61db844470b808963ae5e963ac0 P fc94a650d15247b7b08bd5c3eeaa41e6: Time spent starting tablet: real 0.004s user 0.000s sys 0.005s
I20260620 04:25:06.615576 2019 tablet_bootstrap.cc:654] T 7530c61db844470b808963ae5e963ac0 P 8af8ecc3c97a4bc9908aa7f03122832c: Neither blocks nor log segments found. Creating new log.
I20260620 04:25:06.616379 2019 log.cc:824] T 7530c61db844470b808963ae5e963ac0 P 8af8ecc3c97a4bc9908aa7f03122832c: Log is configured to *not* fsync() on all Append() calls
I20260620 04:25:06.618248 2019 tablet_bootstrap.cc:492] T 7530c61db844470b808963ae5e963ac0 P 8af8ecc3c97a4bc9908aa7f03122832c: No bootstrap required, opened a new log
I20260620 04:25:06.618398 2019 ts_tablet_manager.cc:1397] T 7530c61db844470b808963ae5e963ac0 P 8af8ecc3c97a4bc9908aa7f03122832c: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 04:25:06.621264 2019 raft_consensus.cc:348] T 7530c61db844470b808963ae5e963ac0 P 8af8ecc3c97a4bc9908aa7f03122832c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8af8ecc3c97a4bc9908aa7f03122832c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32803 } } peers { permanent_uuid: "fc94a650d15247b7b08bd5c3eeaa41e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37745 } } peers { permanent_uuid: "92efaefa517c473cadbf761e21859ff4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45753 } }
I20260620 04:25:06.621507 2019 raft_consensus.cc:374] T 7530c61db844470b808963ae5e963ac0 P 8af8ecc3c97a4bc9908aa7f03122832c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:25:06.621563 2019 raft_consensus.cc:729] T 7530c61db844470b808963ae5e963ac0 P 8af8ecc3c97a4bc9908aa7f03122832c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8af8ecc3c97a4bc9908aa7f03122832c, State: Initialized, Role: FOLLOWER
I20260620 04:25:06.621987 2019 consensus_queue.cc:260] T 7530c61db844470b808963ae5e963ac0 P 8af8ecc3c97a4bc9908aa7f03122832c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8af8ecc3c97a4bc9908aa7f03122832c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32803 } } peers { permanent_uuid: "fc94a650d15247b7b08bd5c3eeaa41e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37745 } } peers { permanent_uuid: "92efaefa517c473cadbf761e21859ff4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45753 } }
I20260620 04:25:06.622612 1853 heartbeater.cc:503] Master 127.0.98.62:37763 was elected leader, sending a full tablet report...
I20260620 04:25:06.622964 2019 ts_tablet_manager.cc:1428] T 7530c61db844470b808963ae5e963ac0 P 8af8ecc3c97a4bc9908aa7f03122832c: Time spent starting tablet: real 0.005s user 0.001s sys 0.004s
W20260620 04:25:06.634075 1714 tablet.cc:2369] T 7530c61db844470b808963ae5e963ac0 P 92efaefa517c473cadbf761e21859ff4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 04:25:06.785279 1994 tablet.cc:2369] T 7530c61db844470b808963ae5e963ac0 P fc94a650d15247b7b08bd5c3eeaa41e6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 04:25:06.834038 1854 tablet.cc:2369] T 7530c61db844470b808963ae5e963ac0 P 8af8ecc3c97a4bc9908aa7f03122832c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 04:25:06.950898 2021 raft_consensus.cc:482] T 7530c61db844470b808963ae5e963ac0 P 92efaefa517c473cadbf761e21859ff4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:25:06.951174 2021 raft_consensus.cc:504] T 7530c61db844470b808963ae5e963ac0 P 92efaefa517c473cadbf761e21859ff4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8af8ecc3c97a4bc9908aa7f03122832c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32803 } } peers { permanent_uuid: "fc94a650d15247b7b08bd5c3eeaa41e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37745 } } peers { permanent_uuid: "92efaefa517c473cadbf761e21859ff4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45753 } }
I20260620 04:25:06.952418 2021 leader_election.cc:288] T 7530c61db844470b808963ae5e963ac0 P 92efaefa517c473cadbf761e21859ff4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8af8ecc3c97a4bc9908aa7f03122832c (127.0.98.2:32803), fc94a650d15247b7b08bd5c3eeaa41e6 (127.0.98.3:37745)
I20260620 04:25:06.956075 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:06.952811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39957 (local address 127.0.98.2:32803)
0620 04:25:06.952978 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:06.952982 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:06.953145 (+ 163us) server_negotiation.cc:407] Connection header received
0620 04:25:06.953251 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:06.953253 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:06.953301 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:06.953369 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:25:06.953584 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:06.954087 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:06.955147 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:06.955544 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:06.955637 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:06.955695 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:06.955870 (+ 175us) server_negotiation.cc:299] Negotiation successful
0620 04:25:06.955934 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260620 04:25:06.956079 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:06.952803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37017 (local address 127.0.98.3:37745)
0620 04:25:06.952924 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:06.952929 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:06.953092 (+ 163us) server_negotiation.cc:407] Connection header received
0620 04:25:06.953170 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:06.953174 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:06.953221 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:06.953282 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:25:06.953540 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:06.954053 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:06.955131 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:06.955520 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:06.955604 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:06.955665 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:06.955834 (+ 169us) server_negotiation.cc:299] Negotiation successful
0620 04:25:06.955905 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 04:25:06.956096 2026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:06.952669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32803 (local address 127.0.98.1:39957)
0620 04:25:06.953059 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:06.953071 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:06.953171 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:06.953391 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:06.953393 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:06.953408 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:25:06.953491 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:06.953497 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:06.954189 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:06.954193 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:06.955034 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:06.955043 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:06.955655 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:06.955658 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:06.955760 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:06.955777 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 04:25:06.955848 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 04:25:06.955914 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"mutex_wait_us":140,"thread_start_us":60,"threads_started":1}
I20260620 04:25:06.956108 2025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:06.952666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37745 (local address 127.0.98.1:37017)
0620 04:25:06.952989 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:06.953002 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:06.953093 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:06.953304 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:06.953306 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:06.953320 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:25:06.953438 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:06.953445 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:06.954156 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:06.954159 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:06.955021 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:06.955030 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:06.955636 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:06.955639 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:06.955729 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:06.955751 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 04:25:06.955808 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 04:25:06.955878 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"mutex_wait_us":1,"thread_start_us":84,"threads_started":1}
I20260620 04:25:06.956836 1807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7530c61db844470b808963ae5e963ac0" candidate_uuid: "92efaefa517c473cadbf761e21859ff4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8af8ecc3c97a4bc9908aa7f03122832c" is_pre_election: true
I20260620 04:25:06.956933 1947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7530c61db844470b808963ae5e963ac0" candidate_uuid: "92efaefa517c473cadbf761e21859ff4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc94a650d15247b7b08bd5c3eeaa41e6" is_pre_election: true
I20260620 04:25:06.957130 1807 raft_consensus.cc:2454] T 7530c61db844470b808963ae5e963ac0 P 8af8ecc3c97a4bc9908aa7f03122832c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92efaefa517c473cadbf761e21859ff4 in term 0.
I20260620 04:25:06.957186 1947 raft_consensus.cc:2454] T 7530c61db844470b808963ae5e963ac0 P fc94a650d15247b7b08bd5c3eeaa41e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92efaefa517c473cadbf761e21859ff4 in term 0.
I20260620 04:25:06.957612 1601 leader_election.cc:302] T 7530c61db844470b808963ae5e963ac0 P 92efaefa517c473cadbf761e21859ff4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8af8ecc3c97a4bc9908aa7f03122832c, 92efaefa517c473cadbf761e21859ff4; no voters:
I20260620 04:25:06.957860 2021 raft_consensus.cc:2781] T 7530c61db844470b808963ae5e963ac0 P 92efaefa517c473cadbf761e21859ff4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 04:25:06.957943 2021 raft_consensus.cc:482] T 7530c61db844470b808963ae5e963ac0 P 92efaefa517c473cadbf761e21859ff4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 04:25:06.957989 2021 raft_consensus.cc:3037] T 7530c61db844470b808963ae5e963ac0 P 92efaefa517c473cadbf761e21859ff4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:25:06.958942 2021 raft_consensus.cc:504] T 7530c61db844470b808963ae5e963ac0 P 92efaefa517c473cadbf761e21859ff4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8af8ecc3c97a4bc9908aa7f03122832c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32803 } } peers { permanent_uuid: "fc94a650d15247b7b08bd5c3eeaa41e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37745 } } peers { permanent_uuid: "92efaefa517c473cadbf761e21859ff4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45753 } }
I20260620 04:25:06.959288 2021 leader_election.cc:288] T 7530c61db844470b808963ae5e963ac0 P 92efaefa517c473cadbf761e21859ff4 [CANDIDATE]: Term 1 election: Requested vote from peers 8af8ecc3c97a4bc9908aa7f03122832c (127.0.98.2:32803), fc94a650d15247b7b08bd5c3eeaa41e6 (127.0.98.3:37745)
I20260620 04:25:06.959558 1807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7530c61db844470b808963ae5e963ac0" candidate_uuid: "92efaefa517c473cadbf761e21859ff4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8af8ecc3c97a4bc9908aa7f03122832c"
I20260620 04:25:06.959650 1947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7530c61db844470b808963ae5e963ac0" candidate_uuid: "92efaefa517c473cadbf761e21859ff4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc94a650d15247b7b08bd5c3eeaa41e6"
I20260620 04:25:06.959673 1807 raft_consensus.cc:3037] T 7530c61db844470b808963ae5e963ac0 P 8af8ecc3c97a4bc9908aa7f03122832c [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:25:06.959764 1947 raft_consensus.cc:3037] T 7530c61db844470b808963ae5e963ac0 P fc94a650d15247b7b08bd5c3eeaa41e6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:25:06.960690 1807 raft_consensus.cc:2454] T 7530c61db844470b808963ae5e963ac0 P 8af8ecc3c97a4bc9908aa7f03122832c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92efaefa517c473cadbf761e21859ff4 in term 1.
I20260620 04:25:06.960960 1601 leader_election.cc:302] T 7530c61db844470b808963ae5e963ac0 P 92efaefa517c473cadbf761e21859ff4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8af8ecc3c97a4bc9908aa7f03122832c, 92efaefa517c473cadbf761e21859ff4; no voters:
I20260620 04:25:06.961114 2021 raft_consensus.cc:2781] T 7530c61db844470b808963ae5e963ac0 P 92efaefa517c473cadbf761e21859ff4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:25:06.961141 1947 raft_consensus.cc:2454] T 7530c61db844470b808963ae5e963ac0 P fc94a650d15247b7b08bd5c3eeaa41e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92efaefa517c473cadbf761e21859ff4 in term 1.
I20260620 04:25:06.961265 2021 raft_consensus.cc:686] T 7530c61db844470b808963ae5e963ac0 P 92efaefa517c473cadbf761e21859ff4 [term 1 LEADER]: Becoming Leader. State: Replica: 92efaefa517c473cadbf761e21859ff4, State: Running, Role: LEADER
I20260620 04:25:06.961601 2021 consensus_queue.cc:237] T 7530c61db844470b808963ae5e963ac0 P 92efaefa517c473cadbf761e21859ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8af8ecc3c97a4bc9908aa7f03122832c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32803 } } peers { permanent_uuid: "fc94a650d15247b7b08bd5c3eeaa41e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37745 } } peers { permanent_uuid: "92efaefa517c473cadbf761e21859ff4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45753 } }
I20260620 04:25:06.964550 1510 catalog_manager.cc:5495] T 7530c61db844470b808963ae5e963ac0 P 92efaefa517c473cadbf761e21859ff4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 92efaefa517c473cadbf761e21859ff4 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "92efaefa517c473cadbf761e21859ff4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8af8ecc3c97a4bc9908aa7f03122832c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fc94a650d15247b7b08bd5c3eeaa41e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92efaefa517c473cadbf761e21859ff4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45753 } health_report { overall_health: HEALTHY } } }
I20260620 04:25:07.052784 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:07.049235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43144 (local address 127.0.98.1:45753)
0620 04:25:07.049364 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:07.049368 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:07.049382 (+ 14us) server_negotiation.cc:407] Connection header received
0620 04:25:07.049572 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:07.049575 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:07.049630 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:07.049714 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:25:07.050003 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:07.050519 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:07.051376 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:07.051606 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:07.051685 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:07.052029 (+ 344us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:25:07.052447 (+ 418us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:25:07.052539 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:07.052616 (+ 77us) server_negotiation.cc:299] Negotiation successful
0620 04:25:07.052670 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 04:25:07.058444 1947 raft_consensus.cc:1261] T 7530c61db844470b808963ae5e963ac0 P fc94a650d15247b7b08bd5c3eeaa41e6 [term 1 FOLLOWER]: Refusing update from remote peer 92efaefa517c473cadbf761e21859ff4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:25:07.058480 1807 raft_consensus.cc:1261] T 7530c61db844470b808963ae5e963ac0 P 8af8ecc3c97a4bc9908aa7f03122832c [term 1 FOLLOWER]: Refusing update from remote peer 92efaefa517c473cadbf761e21859ff4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:25:07.059152 2027 consensus_queue.cc:1035] T 7530c61db844470b808963ae5e963ac0 P 92efaefa517c473cadbf761e21859ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8af8ecc3c97a4bc9908aa7f03122832c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:25:07.059348 2021 consensus_queue.cc:1035] T 7530c61db844470b808963ae5e963ac0 P 92efaefa517c473cadbf761e21859ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc94a650d15247b7b08bd5c3eeaa41e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:25:07.067335 2030 mvcc.cc:205] Tried to move back new op lower bound from 7298783260904054784 to 7298783260516831232. Current Snapshot: MvccSnapshot[applied={T|T < 7298783260904054784}]
I20260620 04:25:07.067502 2032 mvcc.cc:205] Tried to move back new op lower bound from 7298783260904054784 to 7298783260516831232. Current Snapshot: MvccSnapshot[applied={T|T < 7298783260904054784}]
I20260620 04:25:07.078625 1510 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50734:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:25:07.078804 1510 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50734:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:25:07.081238 1510 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fb5b3346cacb41ca9314bc02e26d17a7: Sending DeleteTablet for 3 replicas of tablet 7530c61db844470b808963ae5e963ac0
I20260620 04:25:07.082126 1787 tablet_service.cc:1514] Processing DeleteTablet for tablet 7530c61db844470b808963ae5e963ac0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:25:07 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49882
I20260620 04:25:07.082139 1647 tablet_service.cc:1514] Processing DeleteTablet for tablet 7530c61db844470b808963ae5e963ac0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:25:07 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43140
I20260620 04:25:07.082132 1927 tablet_service.cc:1514] Processing DeleteTablet for tablet 7530c61db844470b808963ae5e963ac0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:25:07 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51314
I20260620 04:25:07.082791 2041 tablet_replica.cc:330] T 7530c61db844470b808963ae5e963ac0 P 8af8ecc3c97a4bc9908aa7f03122832c: stopping tablet replica
I20260620 04:25:07.083137 2041 raft_consensus.cc:2229] T 7530c61db844470b808963ae5e963ac0 P 8af8ecc3c97a4bc9908aa7f03122832c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:25:07.083473 2041 raft_consensus.cc:2258] T 7530c61db844470b808963ae5e963ac0 P 8af8ecc3c97a4bc9908aa7f03122832c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:25:07.083585 2042 tablet_replica.cc:330] T 7530c61db844470b808963ae5e963ac0 P 92efaefa517c473cadbf761e21859ff4: stopping tablet replica
I20260620 04:25:07.083933 2043 tablet_replica.cc:330] T 7530c61db844470b808963ae5e963ac0 P fc94a650d15247b7b08bd5c3eeaa41e6: stopping tablet replica
I20260620 04:25:07.083871 1510 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50734:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 04:25:07.084300 2043 raft_consensus.cc:2229] T 7530c61db844470b808963ae5e963ac0 P fc94a650d15247b7b08bd5c3eeaa41e6 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260620 04:25:07.084359 1510 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 04:25:07.084625 2043 raft_consensus.cc:2258] T 7530c61db844470b808963ae5e963ac0 P fc94a650d15247b7b08bd5c3eeaa41e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:25:07.085062 2042 raft_consensus.cc:2229] T 7530c61db844470b808963ae5e963ac0 P 92efaefa517c473cadbf761e21859ff4 [term 1 LEADER]: Raft consensus shutting down.
I20260620 04:25:07.085906 2042 raft_consensus.cc:2258] T 7530c61db844470b808963ae5e963ac0 P 92efaefa517c473cadbf761e21859ff4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:25:07.087545 2041 ts_tablet_manager.cc:1905] T 7530c61db844470b808963ae5e963ac0 P 8af8ecc3c97a4bc9908aa7f03122832c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:25:07.088821 2042 ts_tablet_manager.cc:1905] T 7530c61db844470b808963ae5e963ac0 P 92efaefa517c473cadbf761e21859ff4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:25:07.089771 1787 tablet_service.cc:1467] Processing CreateTablet for tablet 004fe054beb54731a67406bfde665ed5 (DEFAULT_TABLE table=test-table [id=23d3f9a8302042739f1535f154a50705]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:25:07.089776 1647 tablet_service.cc:1467] Processing CreateTablet for tablet 004fe054beb54731a67406bfde665ed5 (DEFAULT_TABLE table=test-table [id=23d3f9a8302042739f1535f154a50705]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:25:07.090030 1787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 004fe054beb54731a67406bfde665ed5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:25:07.090056 1647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 004fe054beb54731a67406bfde665ed5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:25:07.090040 1927 tablet_service.cc:1467] Processing CreateTablet for tablet 004fe054beb54731a67406bfde665ed5 (DEFAULT_TABLE table=test-table [id=23d3f9a8302042739f1535f154a50705]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:25:07.090288 1927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 004fe054beb54731a67406bfde665ed5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:25:07.092335 2019 tablet_bootstrap.cc:492] T 004fe054beb54731a67406bfde665ed5 P 8af8ecc3c97a4bc9908aa7f03122832c: Bootstrap starting.
I20260620 04:25:07.092836 2017 tablet_bootstrap.cc:492] T 004fe054beb54731a67406bfde665ed5 P fc94a650d15247b7b08bd5c3eeaa41e6: Bootstrap starting.
I20260620 04:25:07.093247 2019 tablet_bootstrap.cc:654] T 004fe054beb54731a67406bfde665ed5 P 8af8ecc3c97a4bc9908aa7f03122832c: Neither blocks nor log segments found. Creating new log.
I20260620 04:25:07.093676 2017 tablet_bootstrap.cc:654] T 004fe054beb54731a67406bfde665ed5 P fc94a650d15247b7b08bd5c3eeaa41e6: Neither blocks nor log segments found. Creating new log.
I20260620 04:25:07.093842 2016 tablet_bootstrap.cc:492] T 004fe054beb54731a67406bfde665ed5 P 92efaefa517c473cadbf761e21859ff4: Bootstrap starting.
I20260620 04:25:07.094987 2017 tablet_bootstrap.cc:492] T 004fe054beb54731a67406bfde665ed5 P fc94a650d15247b7b08bd5c3eeaa41e6: No bootstrap required, opened a new log
I20260620 04:25:07.095067 2017 ts_tablet_manager.cc:1397] T 004fe054beb54731a67406bfde665ed5 P fc94a650d15247b7b08bd5c3eeaa41e6: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 04:25:07.095206 2016 tablet_bootstrap.cc:654] T 004fe054beb54731a67406bfde665ed5 P 92efaefa517c473cadbf761e21859ff4: Neither blocks nor log segments found. Creating new log.
I20260620 04:25:07.095463 2017 raft_consensus.cc:348] T 004fe054beb54731a67406bfde665ed5 P fc94a650d15247b7b08bd5c3eeaa41e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92efaefa517c473cadbf761e21859ff4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45753 } } peers { permanent_uuid: "8af8ecc3c97a4bc9908aa7f03122832c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32803 } } peers { permanent_uuid: "fc94a650d15247b7b08bd5c3eeaa41e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37745 } }
I20260620 04:25:07.095564 2017 raft_consensus.cc:374] T 004fe054beb54731a67406bfde665ed5 P fc94a650d15247b7b08bd5c3eeaa41e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:25:07.095595 2017 raft_consensus.cc:729] T 004fe054beb54731a67406bfde665ed5 P fc94a650d15247b7b08bd5c3eeaa41e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc94a650d15247b7b08bd5c3eeaa41e6, State: Initialized, Role: FOLLOWER
I20260620 04:25:07.095712 2017 consensus_queue.cc:260] T 004fe054beb54731a67406bfde665ed5 P fc94a650d15247b7b08bd5c3eeaa41e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92efaefa517c473cadbf761e21859ff4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45753 } } peers { permanent_uuid: "8af8ecc3c97a4bc9908aa7f03122832c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32803 } } peers { permanent_uuid: "fc94a650d15247b7b08bd5c3eeaa41e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37745 } }
I20260620 04:25:07.095973 2017 ts_tablet_manager.cc:1428] T 004fe054beb54731a67406bfde665ed5 P fc94a650d15247b7b08bd5c3eeaa41e6: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 04:25:07.096330 2043 ts_tablet_manager.cc:1905] T 7530c61db844470b808963ae5e963ac0 P fc94a650d15247b7b08bd5c3eeaa41e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:25:07.098250 2041 ts_tablet_manager.cc:1918] T 7530c61db844470b808963ae5e963ac0 P 8af8ecc3c97a4bc9908aa7f03122832c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:25:07.098343 2041 log.cc:1196] T 7530c61db844470b808963ae5e963ac0 P 8af8ecc3c97a4bc9908aa7f03122832c: Deleting WAL directory at /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929496330398-392-0/minicluster-data/ts-1/wal/wals/7530c61db844470b808963ae5e963ac0
I20260620 04:25:07.098629 2041 ts_tablet_manager.cc:1939] T 7530c61db844470b808963ae5e963ac0 P 8af8ecc3c97a4bc9908aa7f03122832c: Deleting consensus metadata
I20260620 04:25:07.099844 1497 catalog_manager.cc:4841] TS 8af8ecc3c97a4bc9908aa7f03122832c (127.0.98.2:32803): tablet 7530c61db844470b808963ae5e963ac0 (table test-table [id=0e866f4cc59141bea5b118a986488e85]) successfully deleted
I20260620 04:25:07.100747 2016 tablet_bootstrap.cc:492] T 004fe054beb54731a67406bfde665ed5 P 92efaefa517c473cadbf761e21859ff4: No bootstrap required, opened a new log
I20260620 04:25:07.100834 2016 ts_tablet_manager.cc:1397] T 004fe054beb54731a67406bfde665ed5 P 92efaefa517c473cadbf761e21859ff4: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260620 04:25:07.101015 2019 tablet_bootstrap.cc:492] T 004fe054beb54731a67406bfde665ed5 P 8af8ecc3c97a4bc9908aa7f03122832c: No bootstrap required, opened a new log
I20260620 04:25:07.101094 2019 ts_tablet_manager.cc:1397] T 004fe054beb54731a67406bfde665ed5 P 8af8ecc3c97a4bc9908aa7f03122832c: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.001s
I20260620 04:25:07.101244 2043 ts_tablet_manager.cc:1918] T 7530c61db844470b808963ae5e963ac0 P fc94a650d15247b7b08bd5c3eeaa41e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:25:07.101326 2043 log.cc:1196] T 7530c61db844470b808963ae5e963ac0 P fc94a650d15247b7b08bd5c3eeaa41e6: Deleting WAL directory at /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929496330398-392-0/minicluster-data/ts-2/wal/wals/7530c61db844470b808963ae5e963ac0
I20260620 04:25:07.101457 2016 raft_consensus.cc:348] T 004fe054beb54731a67406bfde665ed5 P 92efaefa517c473cadbf761e21859ff4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92efaefa517c473cadbf761e21859ff4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45753 } } peers { permanent_uuid: "8af8ecc3c97a4bc9908aa7f03122832c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32803 } } peers { permanent_uuid: "fc94a650d15247b7b08bd5c3eeaa41e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37745 } }
I20260620 04:25:07.101560 2016 raft_consensus.cc:374] T 004fe054beb54731a67406bfde665ed5 P 92efaefa517c473cadbf761e21859ff4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:25:07.101492 2019 raft_consensus.cc:348] T 004fe054beb54731a67406bfde665ed5 P 8af8ecc3c97a4bc9908aa7f03122832c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92efaefa517c473cadbf761e21859ff4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45753 } } peers { permanent_uuid: "8af8ecc3c97a4bc9908aa7f03122832c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32803 } } peers { permanent_uuid: "fc94a650d15247b7b08bd5c3eeaa41e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37745 } }
I20260620 04:25:07.101591 2016 raft_consensus.cc:729] T 004fe054beb54731a67406bfde665ed5 P 92efaefa517c473cadbf761e21859ff4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92efaefa517c473cadbf761e21859ff4, State: Initialized, Role: FOLLOWER
I20260620 04:25:07.101600 2019 raft_consensus.cc:374] T 004fe054beb54731a67406bfde665ed5 P 8af8ecc3c97a4bc9908aa7f03122832c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:25:07.101611 2043 ts_tablet_manager.cc:1939] T 7530c61db844470b808963ae5e963ac0 P fc94a650d15247b7b08bd5c3eeaa41e6: Deleting consensus metadata
I20260620 04:25:07.101630 2019 raft_consensus.cc:729] T 004fe054beb54731a67406bfde665ed5 P 8af8ecc3c97a4bc9908aa7f03122832c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8af8ecc3c97a4bc9908aa7f03122832c, State: Initialized, Role: FOLLOWER
I20260620 04:25:07.101706 2016 consensus_queue.cc:260] T 004fe054beb54731a67406bfde665ed5 P 92efaefa517c473cadbf761e21859ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92efaefa517c473cadbf761e21859ff4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45753 } } peers { permanent_uuid: "8af8ecc3c97a4bc9908aa7f03122832c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32803 } } peers { permanent_uuid: "fc94a650d15247b7b08bd5c3eeaa41e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37745 } }
I20260620 04:25:07.101742 2019 consensus_queue.cc:260] T 004fe054beb54731a67406bfde665ed5 P 8af8ecc3c97a4bc9908aa7f03122832c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92efaefa517c473cadbf761e21859ff4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45753 } } peers { permanent_uuid: "8af8ecc3c97a4bc9908aa7f03122832c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32803 } } peers { permanent_uuid: "fc94a650d15247b7b08bd5c3eeaa41e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37745 } }
I20260620 04:25:07.101955 2016 ts_tablet_manager.cc:1428] T 004fe054beb54731a67406bfde665ed5 P 92efaefa517c473cadbf761e21859ff4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 04:25:07.101971 2019 ts_tablet_manager.cc:1428] T 004fe054beb54731a67406bfde665ed5 P 8af8ecc3c97a4bc9908aa7f03122832c: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260620 04:25:07.103801 2042 ts_tablet_manager.cc:1918] T 7530c61db844470b808963ae5e963ac0 P 92efaefa517c473cadbf761e21859ff4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:25:07.103889 2042 log.cc:1196] T 7530c61db844470b808963ae5e963ac0 P 92efaefa517c473cadbf761e21859ff4: Deleting WAL directory at /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929496330398-392-0/minicluster-data/ts-0/wal/wals/7530c61db844470b808963ae5e963ac0
I20260620 04:25:07.104223 2042 ts_tablet_manager.cc:1939] T 7530c61db844470b808963ae5e963ac0 P 92efaefa517c473cadbf761e21859ff4: Deleting consensus metadata
I20260620 04:25:07.105484 1496 catalog_manager.cc:4841] TS 92efaefa517c473cadbf761e21859ff4 (127.0.98.1:45753): tablet 7530c61db844470b808963ae5e963ac0 (table test-table [id=0e866f4cc59141bea5b118a986488e85]) successfully deleted
I20260620 04:25:07.105834 1495 catalog_manager.cc:4841] TS fc94a650d15247b7b08bd5c3eeaa41e6 (127.0.98.3:37745): tablet 7530c61db844470b808963ae5e963ac0 (table test-table [id=0e866f4cc59141bea5b118a986488e85]) successfully deleted
I20260620 04:25:07.273834 2024 raft_consensus.cc:482] T 004fe054beb54731a67406bfde665ed5 P 8af8ecc3c97a4bc9908aa7f03122832c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:25:07.273983 2024 raft_consensus.cc:504] T 004fe054beb54731a67406bfde665ed5 P 8af8ecc3c97a4bc9908aa7f03122832c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92efaefa517c473cadbf761e21859ff4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45753 } } peers { permanent_uuid: "8af8ecc3c97a4bc9908aa7f03122832c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32803 } } peers { permanent_uuid: "fc94a650d15247b7b08bd5c3eeaa41e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37745 } }
I20260620 04:25:07.275022 2024 leader_election.cc:288] T 004fe054beb54731a67406bfde665ed5 P 8af8ecc3c97a4bc9908aa7f03122832c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92efaefa517c473cadbf761e21859ff4 (127.0.98.1:45753), fc94a650d15247b7b08bd5c3eeaa41e6 (127.0.98.3:37745)
I20260620 04:25:07.280210 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:07.275385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51475 (local address 127.0.98.1:45753)
0620 04:25:07.275528 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:07.275533 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:07.275800 (+ 267us) server_negotiation.cc:407] Connection header received
0620 04:25:07.275944 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:07.275947 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:07.276016 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:07.276105 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:25:07.276468 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:07.277260 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:07.279037 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:07.279639 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:07.279738 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:07.279799 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:07.280004 (+ 205us) server_negotiation.cc:299] Negotiation successful
0620 04:25:07.280085 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 04:25:07.280210 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:07.275408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54801 (local address 127.0.98.3:37745)
0620 04:25:07.275523 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:07.275528 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:07.275800 (+ 272us) server_negotiation.cc:407] Connection header received
0620 04:25:07.275935 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:07.275938 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:07.276016 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:07.276097 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:25:07.276497 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:07.277260 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:07.278883 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:07.279531 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:07.279659 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:07.279735 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:07.280001 (+ 266us) server_negotiation.cc:299] Negotiation successful
0620 04:25:07.280080 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 04:25:07.280252 2047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:07.275309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37745 (local address 127.0.98.2:54801)
0620 04:25:07.275671 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:07.275685 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:07.275824 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:07.276128 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:07.276131 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:07.276156 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:25:07.276361 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:07.276367 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:07.277363 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:07.277366 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:07.278736 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:07.278747 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:07.279673 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:07.279676 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:07.279786 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:07.279849 (+ 63us) client_negotiation.cc:769] Sending connection context
0620 04:25:07.279926 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 04:25:07.280059 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"mutex_wait_us":95,"spinlock_wait_cycles":21888,"thread_start_us":97,"threads_started":1}
I20260620 04:25:07.280252 2048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:07.275333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45753 (local address 127.0.98.2:51475)
0620 04:25:07.275684 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:07.275695 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:07.275824 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:07.276128 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:07.276131 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:07.276156 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:25:07.276354 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:07.276361 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:07.277363 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:07.277366 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:07.278911 (+ 1545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:07.278918 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:07.279741 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:07.279744 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:07.279851 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:07.279868 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 04:25:07.279927 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 04:25:07.280059 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"mutex_wait_us":85,"spinlock_wait_cycles":20096,"thread_start_us":167,"threads_started":1}
I20260620 04:25:07.280694 1947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "004fe054beb54731a67406bfde665ed5" candidate_uuid: "8af8ecc3c97a4bc9908aa7f03122832c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc94a650d15247b7b08bd5c3eeaa41e6" is_pre_election: true
I20260620 04:25:07.280833 1947 raft_consensus.cc:2454] T 004fe054beb54731a67406bfde665ed5 P fc94a650d15247b7b08bd5c3eeaa41e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8af8ecc3c97a4bc9908aa7f03122832c in term 0.
I20260620 04:25:07.281177 1739 leader_election.cc:302] T 004fe054beb54731a67406bfde665ed5 P 8af8ecc3c97a4bc9908aa7f03122832c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8af8ecc3c97a4bc9908aa7f03122832c, fc94a650d15247b7b08bd5c3eeaa41e6; no voters:
I20260620 04:25:07.281217 1667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "004fe054beb54731a67406bfde665ed5" candidate_uuid: "8af8ecc3c97a4bc9908aa7f03122832c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92efaefa517c473cadbf761e21859ff4" is_pre_election: true
I20260620 04:25:07.281448 1667 raft_consensus.cc:2454] T 004fe054beb54731a67406bfde665ed5 P 92efaefa517c473cadbf761e21859ff4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8af8ecc3c97a4bc9908aa7f03122832c in term 0.
I20260620 04:25:07.281486 2024 raft_consensus.cc:2781] T 004fe054beb54731a67406bfde665ed5 P 8af8ecc3c97a4bc9908aa7f03122832c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 04:25:07.281540 2024 raft_consensus.cc:482] T 004fe054beb54731a67406bfde665ed5 P 8af8ecc3c97a4bc9908aa7f03122832c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 04:25:07.281610 2024 raft_consensus.cc:3037] T 004fe054beb54731a67406bfde665ed5 P 8af8ecc3c97a4bc9908aa7f03122832c [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:25:07.282552 2024 raft_consensus.cc:504] T 004fe054beb54731a67406bfde665ed5 P 8af8ecc3c97a4bc9908aa7f03122832c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92efaefa517c473cadbf761e21859ff4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45753 } } peers { permanent_uuid: "8af8ecc3c97a4bc9908aa7f03122832c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32803 } } peers { permanent_uuid: "fc94a650d15247b7b08bd5c3eeaa41e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37745 } }
I20260620 04:25:07.282866 2024 leader_election.cc:288] T 004fe054beb54731a67406bfde665ed5 P 8af8ecc3c97a4bc9908aa7f03122832c [CANDIDATE]: Term 1 election: Requested vote from peers 92efaefa517c473cadbf761e21859ff4 (127.0.98.1:45753), fc94a650d15247b7b08bd5c3eeaa41e6 (127.0.98.3:37745)
I20260620 04:25:07.283200 1667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "004fe054beb54731a67406bfde665ed5" candidate_uuid: "8af8ecc3c97a4bc9908aa7f03122832c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92efaefa517c473cadbf761e21859ff4"
I20260620 04:25:07.283316 1667 raft_consensus.cc:3037] T 004fe054beb54731a67406bfde665ed5 P 92efaefa517c473cadbf761e21859ff4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:25:07.283342 1947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "004fe054beb54731a67406bfde665ed5" candidate_uuid: "8af8ecc3c97a4bc9908aa7f03122832c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc94a650d15247b7b08bd5c3eeaa41e6"
I20260620 04:25:07.283445 1947 raft_consensus.cc:3037] T 004fe054beb54731a67406bfde665ed5 P fc94a650d15247b7b08bd5c3eeaa41e6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:25:07.284492 1667 raft_consensus.cc:2454] T 004fe054beb54731a67406bfde665ed5 P 92efaefa517c473cadbf761e21859ff4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8af8ecc3c97a4bc9908aa7f03122832c in term 1.
I20260620 04:25:07.284492 1947 raft_consensus.cc:2454] T 004fe054beb54731a67406bfde665ed5 P fc94a650d15247b7b08bd5c3eeaa41e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8af8ecc3c97a4bc9908aa7f03122832c in term 1.
I20260620 04:25:07.284752 1739 leader_election.cc:302] T 004fe054beb54731a67406bfde665ed5 P 8af8ecc3c97a4bc9908aa7f03122832c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8af8ecc3c97a4bc9908aa7f03122832c, fc94a650d15247b7b08bd5c3eeaa41e6; no voters:
I20260620 04:25:07.284914 2024 raft_consensus.cc:2781] T 004fe054beb54731a67406bfde665ed5 P 8af8ecc3c97a4bc9908aa7f03122832c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:25:07.285101 2024 raft_consensus.cc:686] T 004fe054beb54731a67406bfde665ed5 P 8af8ecc3c97a4bc9908aa7f03122832c [term 1 LEADER]: Becoming Leader. State: Replica: 8af8ecc3c97a4bc9908aa7f03122832c, State: Running, Role: LEADER
I20260620 04:25:07.285336 2024 consensus_queue.cc:237] T 004fe054beb54731a67406bfde665ed5 P 8af8ecc3c97a4bc9908aa7f03122832c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92efaefa517c473cadbf761e21859ff4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45753 } } peers { permanent_uuid: "8af8ecc3c97a4bc9908aa7f03122832c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32803 } } peers { permanent_uuid: "fc94a650d15247b7b08bd5c3eeaa41e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37745 } }
I20260620 04:25:07.287544 1509 catalog_manager.cc:5495] T 004fe054beb54731a67406bfde665ed5 P 8af8ecc3c97a4bc9908aa7f03122832c reported cstate change: term changed from 0 to 1, leader changed from <none> to 8af8ecc3c97a4bc9908aa7f03122832c (127.0.98.2). New cstate: current_term: 1 leader_uuid: "8af8ecc3c97a4bc9908aa7f03122832c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92efaefa517c473cadbf761e21859ff4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8af8ecc3c97a4bc9908aa7f03122832c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32803 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fc94a650d15247b7b08bd5c3eeaa41e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37745 } health_report { overall_health: UNKNOWN } } }
Jun 20 04:25:07 dist-test-slave-nk6d krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929505, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:25:07.357622 2056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:07.349503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37763 (local address 127.0.0.1:50740)
0620 04:25:07.349777 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:07.349790 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:07.349880 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:07.350485 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:07.350487 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:07.350859 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:07.350952 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:07.350958 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:07.351697 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:07.351700 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:07.352316 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:07.352324 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:07.352767 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:07.352769 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:07.352826 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:07.353343 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:07.353354 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:07.354684 (+ 1330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:25:07.356222 (+ 1538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:07.356230 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:07.356235 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:07.356447 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:07.356744 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:07.356746 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:07.356748 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:07.356820 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:07.357166 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:25:07.357171 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:25:07.357277 (+ 106us) client_negotiation.cc:769] Sending connection context
0620 04:25:07.357359 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 04:25:07.357474 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":99,"threads_started":1}
I20260620 04:25:07.357676 2057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:07.349567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50740 (local address 127.0.98.62:37763)
0620 04:25:07.349842 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:07.349848 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:07.349862 (+ 14us) server_negotiation.cc:407] Connection header received
0620 04:25:07.349960 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:07.349963 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:07.350002 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:07.350498 (+ 496us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:07.351057 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:07.351607 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:07.352427 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:07.352682 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:07.352760 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:07.354817 (+ 2057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:07.354820 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:07.354822 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:25:07.354848 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:07.356139 (+ 1291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:07.356555 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:07.356558 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:07.356560 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:07.356657 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:07.356904 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:07.356907 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:07.356908 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:07.357089 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:07.357164 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:07.357414 (+ 250us) server_negotiation.cc:299] Negotiation successful
0620 04:25:07.357549 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":112,"threads_started":1}
I20260620 04:25:07.361479 1509 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50740:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 04:25:07.362187 1509 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 04:25:07.366753 1647 tablet_service.cc:1467] Processing CreateTablet for tablet 339052753dd341cdaf21ea78446b6eda (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3cc85832f0d546e5ac9db47d9f3c05b5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 04:25:07.366827 1787 tablet_service.cc:1467] Processing CreateTablet for tablet 339052753dd341cdaf21ea78446b6eda (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3cc85832f0d546e5ac9db47d9f3c05b5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 04:25:07.366942 1927 tablet_service.cc:1467] Processing CreateTablet for tablet 339052753dd341cdaf21ea78446b6eda (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3cc85832f0d546e5ac9db47d9f3c05b5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 04:25:07.367173 1787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 339052753dd341cdaf21ea78446b6eda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:25:07.367194 1647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 339052753dd341cdaf21ea78446b6eda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:25:07.367276 1927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 339052753dd341cdaf21ea78446b6eda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:25:07.369676 2019 tablet_bootstrap.cc:492] T 339052753dd341cdaf21ea78446b6eda P 8af8ecc3c97a4bc9908aa7f03122832c: Bootstrap starting.
I20260620 04:25:07.369817 2016 tablet_bootstrap.cc:492] T 339052753dd341cdaf21ea78446b6eda P 92efaefa517c473cadbf761e21859ff4: Bootstrap starting.
I20260620 04:25:07.369850 2017 tablet_bootstrap.cc:492] T 339052753dd341cdaf21ea78446b6eda P fc94a650d15247b7b08bd5c3eeaa41e6: Bootstrap starting.
I20260620 04:25:07.370419 2019 tablet_bootstrap.cc:654] T 339052753dd341cdaf21ea78446b6eda P 8af8ecc3c97a4bc9908aa7f03122832c: Neither blocks nor log segments found. Creating new log.
I20260620 04:25:07.370625 2016 tablet_bootstrap.cc:654] T 339052753dd341cdaf21ea78446b6eda P 92efaefa517c473cadbf761e21859ff4: Neither blocks nor log segments found. Creating new log.
I20260620 04:25:07.370661 2017 tablet_bootstrap.cc:654] T 339052753dd341cdaf21ea78446b6eda P fc94a650d15247b7b08bd5c3eeaa41e6: Neither blocks nor log segments found. Creating new log.
I20260620 04:25:07.371511 2019 tablet_bootstrap.cc:492] T 339052753dd341cdaf21ea78446b6eda P 8af8ecc3c97a4bc9908aa7f03122832c: No bootstrap required, opened a new log
I20260620 04:25:07.371587 2019 ts_tablet_manager.cc:1397] T 339052753dd341cdaf21ea78446b6eda P 8af8ecc3c97a4bc9908aa7f03122832c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260620 04:25:07.372093 2017 tablet_bootstrap.cc:492] T 339052753dd341cdaf21ea78446b6eda P fc94a650d15247b7b08bd5c3eeaa41e6: No bootstrap required, opened a new log
I20260620 04:25:07.372045 2019 raft_consensus.cc:348] T 339052753dd341cdaf21ea78446b6eda P 8af8ecc3c97a4bc9908aa7f03122832c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92efaefa517c473cadbf761e21859ff4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45753 } } peers { permanent_uuid: "8af8ecc3c97a4bc9908aa7f03122832c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32803 } } peers { permanent_uuid: "fc94a650d15247b7b08bd5c3eeaa41e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37745 } }
I20260620 04:25:07.372166 2019 raft_consensus.cc:374] T 339052753dd341cdaf21ea78446b6eda P 8af8ecc3c97a4bc9908aa7f03122832c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:25:07.372174 2017 ts_tablet_manager.cc:1397] T 339052753dd341cdaf21ea78446b6eda P fc94a650d15247b7b08bd5c3eeaa41e6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 04:25:07.372205 2019 raft_consensus.cc:729] T 339052753dd341cdaf21ea78446b6eda P 8af8ecc3c97a4bc9908aa7f03122832c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8af8ecc3c97a4bc9908aa7f03122832c, State: Initialized, Role: FOLLOWER
I20260620 04:25:07.372330 2019 consensus_queue.cc:260] T 339052753dd341cdaf21ea78446b6eda P 8af8ecc3c97a4bc9908aa7f03122832c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92efaefa517c473cadbf761e21859ff4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45753 } } peers { permanent_uuid: "8af8ecc3c97a4bc9908aa7f03122832c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32803 } } peers { permanent_uuid: "fc94a650d15247b7b08bd5c3eeaa41e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37745 } }
I20260620 04:25:07.372579 2016 tablet_bootstrap.cc:492] T 339052753dd341cdaf21ea78446b6eda P 92efaefa517c473cadbf761e21859ff4: No bootstrap required, opened a new log
I20260620 04:25:07.372587 2019 ts_tablet_manager.cc:1428] T 339052753dd341cdaf21ea78446b6eda P 8af8ecc3c97a4bc9908aa7f03122832c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 04:25:07.372560 2017 raft_consensus.cc:348] T 339052753dd341cdaf21ea78446b6eda P fc94a650d15247b7b08bd5c3eeaa41e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92efaefa517c473cadbf761e21859ff4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45753 } } peers { permanent_uuid: "8af8ecc3c97a4bc9908aa7f03122832c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32803 } } peers { permanent_uuid: "fc94a650d15247b7b08bd5c3eeaa41e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37745 } }
I20260620 04:25:07.372653 2016 ts_tablet_manager.cc:1397] T 339052753dd341cdaf21ea78446b6eda P 92efaefa517c473cadbf761e21859ff4: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 04:25:07.372671 2017 raft_consensus.cc:374] T 339052753dd341cdaf21ea78446b6eda P fc94a650d15247b7b08bd5c3eeaa41e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:25:07.372702 2017 raft_consensus.cc:729] T 339052753dd341cdaf21ea78446b6eda P fc94a650d15247b7b08bd5c3eeaa41e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc94a650d15247b7b08bd5c3eeaa41e6, State: Initialized, Role: FOLLOWER
I20260620 04:25:07.372766 2024 tablet_replica.cc:439] T 339052753dd341cdaf21ea78446b6eda P 8af8ecc3c97a4bc9908aa7f03122832c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92efaefa517c473cadbf761e21859ff4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45753 } } peers { permanent_uuid: "8af8ecc3c97a4bc9908aa7f03122832c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32803 } } peers { permanent_uuid: "fc94a650d15247b7b08bd5c3eeaa41e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37745 } } }
I20260620 04:25:07.372821 2017 consensus_queue.cc:260] T 339052753dd341cdaf21ea78446b6eda P fc94a650d15247b7b08bd5c3eeaa41e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92efaefa517c473cadbf761e21859ff4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45753 } } peers { permanent_uuid: "8af8ecc3c97a4bc9908aa7f03122832c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32803 } } peers { permanent_uuid: "fc94a650d15247b7b08bd5c3eeaa41e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37745 } }
I20260620 04:25:07.372886 2024 tablet_replica.cc:442] T 339052753dd341cdaf21ea78446b6eda P 8af8ecc3c97a4bc9908aa7f03122832c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 04:25:07.373055 2017 ts_tablet_manager.cc:1428] T 339052753dd341cdaf21ea78446b6eda P fc94a650d15247b7b08bd5c3eeaa41e6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 04:25:07.373020 2016 raft_consensus.cc:348] T 339052753dd341cdaf21ea78446b6eda P 92efaefa517c473cadbf761e21859ff4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92efaefa517c473cadbf761e21859ff4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45753 } } peers { permanent_uuid: "8af8ecc3c97a4bc9908aa7f03122832c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32803 } } peers { permanent_uuid: "fc94a650d15247b7b08bd5c3eeaa41e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37745 } }
I20260620 04:25:07.373116 2016 raft_consensus.cc:374] T 339052753dd341cdaf21ea78446b6eda P 92efaefa517c473cadbf761e21859ff4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:25:07.373149 2016 raft_consensus.cc:729] T 339052753dd341cdaf21ea78446b6eda P 92efaefa517c473cadbf761e21859ff4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92efaefa517c473cadbf761e21859ff4, State: Initialized, Role: FOLLOWER
I20260620 04:25:07.373232 2022 tablet_replica.cc:439] T 339052753dd341cdaf21ea78446b6eda P fc94a650d15247b7b08bd5c3eeaa41e6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92efaefa517c473cadbf761e21859ff4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45753 } } peers { permanent_uuid: "8af8ecc3c97a4bc9908aa7f03122832c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32803 } } peers { permanent_uuid: "fc94a650d15247b7b08bd5c3eeaa41e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37745 } } }
I20260620 04:25:07.373258 2016 consensus_queue.cc:260] T 339052753dd341cdaf21ea78446b6eda P 92efaefa517c473cadbf761e21859ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92efaefa517c473cadbf761e21859ff4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45753 } } peers { permanent_uuid: "8af8ecc3c97a4bc9908aa7f03122832c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32803 } } peers { permanent_uuid: "fc94a650d15247b7b08bd5c3eeaa41e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37745 } }
I20260620 04:25:07.373353 2022 tablet_replica.cc:442] T 339052753dd341cdaf21ea78446b6eda P fc94a650d15247b7b08bd5c3eeaa41e6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 04:25:07.373503 2035 tablet_replica.cc:439] T 339052753dd341cdaf21ea78446b6eda P 92efaefa517c473cadbf761e21859ff4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92efaefa517c473cadbf761e21859ff4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45753 } } peers { permanent_uuid: "8af8ecc3c97a4bc9908aa7f03122832c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32803 } } peers { permanent_uuid: "fc94a650d15247b7b08bd5c3eeaa41e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37745 } } }
I20260620 04:25:07.373646 2035 tablet_replica.cc:442] T 339052753dd341cdaf21ea78446b6eda P 92efaefa517c473cadbf761e21859ff4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 04:25:07.373519 2016 ts_tablet_manager.cc:1428] T 339052753dd341cdaf21ea78446b6eda P 92efaefa517c473cadbf761e21859ff4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 04:25:07.501978 2024 raft_consensus.cc:482] T 339052753dd341cdaf21ea78446b6eda P 8af8ecc3c97a4bc9908aa7f03122832c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:25:07.502137 2024 raft_consensus.cc:504] T 339052753dd341cdaf21ea78446b6eda P 8af8ecc3c97a4bc9908aa7f03122832c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92efaefa517c473cadbf761e21859ff4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45753 } } peers { permanent_uuid: "8af8ecc3c97a4bc9908aa7f03122832c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32803 } } peers { permanent_uuid: "fc94a650d15247b7b08bd5c3eeaa41e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37745 } }
I20260620 04:25:07.502538 2024 leader_election.cc:288] T 339052753dd341cdaf21ea78446b6eda P 8af8ecc3c97a4bc9908aa7f03122832c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92efaefa517c473cadbf761e21859ff4 (127.0.98.1:45753), fc94a650d15247b7b08bd5c3eeaa41e6 (127.0.98.3:37745)
I20260620 04:25:07.503005 1947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "339052753dd341cdaf21ea78446b6eda" candidate_uuid: "8af8ecc3c97a4bc9908aa7f03122832c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc94a650d15247b7b08bd5c3eeaa41e6" is_pre_election: true
I20260620 04:25:07.503006 1667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "339052753dd341cdaf21ea78446b6eda" candidate_uuid: "8af8ecc3c97a4bc9908aa7f03122832c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92efaefa517c473cadbf761e21859ff4" is_pre_election: true
I20260620 04:25:07.503134 1947 raft_consensus.cc:2454] T 339052753dd341cdaf21ea78446b6eda P fc94a650d15247b7b08bd5c3eeaa41e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8af8ecc3c97a4bc9908aa7f03122832c in term 0.
I20260620 04:25:07.503168 1667 raft_consensus.cc:2454] T 339052753dd341cdaf21ea78446b6eda P 92efaefa517c473cadbf761e21859ff4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8af8ecc3c97a4bc9908aa7f03122832c in term 0.
I20260620 04:25:07.503425 1739 leader_election.cc:302] T 339052753dd341cdaf21ea78446b6eda P 8af8ecc3c97a4bc9908aa7f03122832c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8af8ecc3c97a4bc9908aa7f03122832c, fc94a650d15247b7b08bd5c3eeaa41e6; no voters:
I20260620 04:25:07.503603 2024 raft_consensus.cc:2781] T 339052753dd341cdaf21ea78446b6eda P 8af8ecc3c97a4bc9908aa7f03122832c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 04:25:07.503660 2024 raft_consensus.cc:482] T 339052753dd341cdaf21ea78446b6eda P 8af8ecc3c97a4bc9908aa7f03122832c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 04:25:07.503690 2024 raft_consensus.cc:3037] T 339052753dd341cdaf21ea78446b6eda P 8af8ecc3c97a4bc9908aa7f03122832c [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:25:07.504685 2024 raft_consensus.cc:504] T 339052753dd341cdaf21ea78446b6eda P 8af8ecc3c97a4bc9908aa7f03122832c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92efaefa517c473cadbf761e21859ff4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45753 } } peers { permanent_uuid: "8af8ecc3c97a4bc9908aa7f03122832c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32803 } } peers { permanent_uuid: "fc94a650d15247b7b08bd5c3eeaa41e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37745 } }
I20260620 04:25:07.504992 2024 leader_election.cc:288] T 339052753dd341cdaf21ea78446b6eda P 8af8ecc3c97a4bc9908aa7f03122832c [CANDIDATE]: Term 1 election: Requested vote from peers 92efaefa517c473cadbf761e21859ff4 (127.0.98.1:45753), fc94a650d15247b7b08bd5c3eeaa41e6 (127.0.98.3:37745)
I20260620 04:25:07.505340 1667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "339052753dd341cdaf21ea78446b6eda" candidate_uuid: "8af8ecc3c97a4bc9908aa7f03122832c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92efaefa517c473cadbf761e21859ff4"
I20260620 04:25:07.505357 1947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "339052753dd341cdaf21ea78446b6eda" candidate_uuid: "8af8ecc3c97a4bc9908aa7f03122832c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc94a650d15247b7b08bd5c3eeaa41e6"
I20260620 04:25:07.505465 1667 raft_consensus.cc:3037] T 339052753dd341cdaf21ea78446b6eda P 92efaefa517c473cadbf761e21859ff4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:25:07.505475 1947 raft_consensus.cc:3037] T 339052753dd341cdaf21ea78446b6eda P fc94a650d15247b7b08bd5c3eeaa41e6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:25:07.506578 1667 raft_consensus.cc:2454] T 339052753dd341cdaf21ea78446b6eda P 92efaefa517c473cadbf761e21859ff4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8af8ecc3c97a4bc9908aa7f03122832c in term 1.
I20260620 04:25:07.506578 1947 raft_consensus.cc:2454] T 339052753dd341cdaf21ea78446b6eda P fc94a650d15247b7b08bd5c3eeaa41e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8af8ecc3c97a4bc9908aa7f03122832c in term 1.
I20260620 04:25:07.506827 1739 leader_election.cc:302] T 339052753dd341cdaf21ea78446b6eda P 8af8ecc3c97a4bc9908aa7f03122832c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8af8ecc3c97a4bc9908aa7f03122832c, fc94a650d15247b7b08bd5c3eeaa41e6; no voters:
I20260620 04:25:07.506970 2024 raft_consensus.cc:2781] T 339052753dd341cdaf21ea78446b6eda P 8af8ecc3c97a4bc9908aa7f03122832c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:25:07.507051 2024 raft_consensus.cc:686] T 339052753dd341cdaf21ea78446b6eda P 8af8ecc3c97a4bc9908aa7f03122832c [term 1 LEADER]: Becoming Leader. State: Replica: 8af8ecc3c97a4bc9908aa7f03122832c, State: Running, Role: LEADER
I20260620 04:25:07.507184 2024 consensus_queue.cc:237] T 339052753dd341cdaf21ea78446b6eda P 8af8ecc3c97a4bc9908aa7f03122832c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92efaefa517c473cadbf761e21859ff4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45753 } } peers { permanent_uuid: "8af8ecc3c97a4bc9908aa7f03122832c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32803 } } peers { permanent_uuid: "fc94a650d15247b7b08bd5c3eeaa41e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37745 } }
I20260620 04:25:07.507781 2049 tablet_replica.cc:439] T 339052753dd341cdaf21ea78446b6eda P 8af8ecc3c97a4bc9908aa7f03122832c: TxnStatusTablet state changed. Reason: New leader 8af8ecc3c97a4bc9908aa7f03122832c. Latest consensus state: current_term: 1 leader_uuid: "8af8ecc3c97a4bc9908aa7f03122832c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92efaefa517c473cadbf761e21859ff4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45753 } } peers { permanent_uuid: "8af8ecc3c97a4bc9908aa7f03122832c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32803 } } peers { permanent_uuid: "fc94a650d15247b7b08bd5c3eeaa41e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37745 } } }
I20260620 04:25:07.507936 2049 tablet_replica.cc:442] T 339052753dd341cdaf21ea78446b6eda P 8af8ecc3c97a4bc9908aa7f03122832c: This TxnStatusTablet replica's current role is: LEADER
I20260620 04:25:07.508245 2062 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 04:25:07.508402 2062 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 04:25:07.508880 1507 catalog_manager.cc:5495] T 339052753dd341cdaf21ea78446b6eda P 8af8ecc3c97a4bc9908aa7f03122832c reported cstate change: term changed from 0 to 1, leader changed from <none> to 8af8ecc3c97a4bc9908aa7f03122832c (127.0.98.2). New cstate: current_term: 1 leader_uuid: "8af8ecc3c97a4bc9908aa7f03122832c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92efaefa517c473cadbf761e21859ff4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8af8ecc3c97a4bc9908aa7f03122832c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32803 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fc94a650d15247b7b08bd5c3eeaa41e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37745 } health_report { overall_health: UNKNOWN } } }
I20260620 04:25:07.530088 2056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:07.526142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32803 (local address 127.0.0.1:49884)
0620 04:25:07.526318 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:07.526336 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:07.526475 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:07.526870 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:07.526873 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:07.526883 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 04:25:07.526992 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:07.527000 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:07.527786 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:07.527789 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:07.528483 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:07.528493 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:07.528951 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:07.528955 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:07.529046 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:07.529093 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 04:25:07.529773 (+ 680us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 04:25:07.529788 (+ 15us) client_negotiation.cc:769] Sending connection context
0620 04:25:07.529859 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 04:25:07.529951 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260620 04:25:07.530105 2065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:07.526295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49884 (local address 127.0.98.2:32803)
0620 04:25:07.526601 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:07.526611 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:07.526626 (+ 15us) server_negotiation.cc:407] Connection header received
0620 04:25:07.526692 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:07.526694 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:07.526747 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:07.526849 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:25:07.527116 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:07.527673 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:07.528603 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:07.528846 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:07.528951 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:07.529254 (+ 303us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:25:07.529676 (+ 422us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:25:07.529786 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:07.529869 (+ 83us) server_negotiation.cc:299] Negotiation successful
0620 04:25:07.529933 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":112,"threads_started":1}
I20260620 04:25:07.533018 1947 raft_consensus.cc:1261] T 339052753dd341cdaf21ea78446b6eda P fc94a650d15247b7b08bd5c3eeaa41e6 [term 1 FOLLOWER]: Refusing update from remote peer 8af8ecc3c97a4bc9908aa7f03122832c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:25:07.533201 1667 raft_consensus.cc:1261] T 339052753dd341cdaf21ea78446b6eda P 92efaefa517c473cadbf761e21859ff4 [term 1 FOLLOWER]: Refusing update from remote peer 8af8ecc3c97a4bc9908aa7f03122832c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:25:07.533521 2049 consensus_queue.cc:1035] T 339052753dd341cdaf21ea78446b6eda P 8af8ecc3c97a4bc9908aa7f03122832c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc94a650d15247b7b08bd5c3eeaa41e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:25:07.533860 2024 consensus_queue.cc:1035] T 339052753dd341cdaf21ea78446b6eda P 8af8ecc3c97a4bc9908aa7f03122832c [LEADER]: Connected to new peer: Peer: permanent_uuid: "92efaefa517c473cadbf761e21859ff4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:25:07.534905 2022 tablet_replica.cc:439] T 339052753dd341cdaf21ea78446b6eda P fc94a650d15247b7b08bd5c3eeaa41e6: TxnStatusTablet state changed. Reason: New leader 8af8ecc3c97a4bc9908aa7f03122832c. Latest consensus state: current_term: 1 leader_uuid: "8af8ecc3c97a4bc9908aa7f03122832c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92efaefa517c473cadbf761e21859ff4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45753 } } peers { permanent_uuid: "8af8ecc3c97a4bc9908aa7f03122832c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32803 } } peers { permanent_uuid: "fc94a650d15247b7b08bd5c3eeaa41e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37745 } } }
I20260620 04:25:07.535008 2022 tablet_replica.cc:442] T 339052753dd341cdaf21ea78446b6eda P fc94a650d15247b7b08bd5c3eeaa41e6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 04:25:07.537387 2035 tablet_replica.cc:439] T 339052753dd341cdaf21ea78446b6eda P 92efaefa517c473cadbf761e21859ff4: TxnStatusTablet state changed. Reason: New leader 8af8ecc3c97a4bc9908aa7f03122832c. Latest consensus state: current_term: 1 leader_uuid: "8af8ecc3c97a4bc9908aa7f03122832c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92efaefa517c473cadbf761e21859ff4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45753 } } peers { permanent_uuid: "8af8ecc3c97a4bc9908aa7f03122832c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32803 } } peers { permanent_uuid: "fc94a650d15247b7b08bd5c3eeaa41e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37745 } } }
I20260620 04:25:07.537495 2035 tablet_replica.cc:442] T 339052753dd341cdaf21ea78446b6eda P 92efaefa517c473cadbf761e21859ff4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 04:25:07.539152 2049 tablet_replica.cc:439] T 339052753dd341cdaf21ea78446b6eda P 8af8ecc3c97a4bc9908aa7f03122832c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8af8ecc3c97a4bc9908aa7f03122832c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92efaefa517c473cadbf761e21859ff4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45753 } } peers { permanent_uuid: "8af8ecc3c97a4bc9908aa7f03122832c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32803 } } peers { permanent_uuid: "fc94a650d15247b7b08bd5c3eeaa41e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37745 } } }
I20260620 04:25:07.539269 2049 tablet_replica.cc:442] T 339052753dd341cdaf21ea78446b6eda P 8af8ecc3c97a4bc9908aa7f03122832c: This TxnStatusTablet replica's current role is: LEADER
I20260620 04:25:07.540350 2022 tablet_replica.cc:439] T 339052753dd341cdaf21ea78446b6eda P fc94a650d15247b7b08bd5c3eeaa41e6: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8af8ecc3c97a4bc9908aa7f03122832c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92efaefa517c473cadbf761e21859ff4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45753 } } peers { permanent_uuid: "8af8ecc3c97a4bc9908aa7f03122832c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32803 } } peers { permanent_uuid: "fc94a650d15247b7b08bd5c3eeaa41e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37745 } } }
I20260620 04:25:07.540467 2022 tablet_replica.cc:442] T 339052753dd341cdaf21ea78446b6eda P fc94a650d15247b7b08bd5c3eeaa41e6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 04:25:07.551859 2065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:07.548834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49890 (local address 127.0.98.2:32803)
0620 04:25:07.549159 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:07.549163 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:07.549172 (+ 9us) server_negotiation.cc:407] Connection header received
0620 04:25:07.549205 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:07.549207 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:07.549244 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:07.549303 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:25:07.549534 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:07.550104 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:07.550874 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:07.551141 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:07.551224 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:07.551461 (+ 237us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:25:07.551533 (+ 72us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:25:07.551589 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:07.551679 (+ 90us) server_negotiation.cc:299] Negotiation successful
0620 04:25:07.551727 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258}
I20260620 04:25:07.553273 2024 tablet_replica.cc:439] T 339052753dd341cdaf21ea78446b6eda P 8af8ecc3c97a4bc9908aa7f03122832c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8af8ecc3c97a4bc9908aa7f03122832c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92efaefa517c473cadbf761e21859ff4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45753 } } peers { permanent_uuid: "8af8ecc3c97a4bc9908aa7f03122832c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32803 } } peers { permanent_uuid: "fc94a650d15247b7b08bd5c3eeaa41e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37745 } } }
I20260620 04:25:07.553393 2024 tablet_replica.cc:442] T 339052753dd341cdaf21ea78446b6eda P 8af8ecc3c97a4bc9908aa7f03122832c: This TxnStatusTablet replica's current role is: LEADER
I20260620 04:25:07.554893 2035 tablet_replica.cc:439] T 339052753dd341cdaf21ea78446b6eda P 92efaefa517c473cadbf761e21859ff4: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8af8ecc3c97a4bc9908aa7f03122832c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92efaefa517c473cadbf761e21859ff4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45753 } } peers { permanent_uuid: "8af8ecc3c97a4bc9908aa7f03122832c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32803 } } peers { permanent_uuid: "fc94a650d15247b7b08bd5c3eeaa41e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37745 } } }
I20260620 04:25:07.555003 2035 tablet_replica.cc:442] T 339052753dd341cdaf21ea78446b6eda P 92efaefa517c473cadbf761e21859ff4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 04:25:07.562038 2070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:07.558869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32803 (local address 127.0.98.2:57865)
0620 04:25:07.559136 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:07.559147 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:07.559237 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:07.559421 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:07.559424 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:07.559430 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 04:25:07.559536 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:07.559540 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:07.560238 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:07.560241 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:07.560863 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:07.560869 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:07.561425 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:07.561427 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:07.561509 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:07.561546 (+ 37us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 04:25:07.561787 (+ 241us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 04:25:07.561791 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 04:25:07.561829 (+ 38us) client_negotiation.cc:240] Negotiation successful
0620 04:25:07.561884 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"mutex_wait_us":1,"thread_start_us":101,"threads_started":1}
I20260620 04:25:07.562059 2065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:07.558911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57865 (local address 127.0.98.2:32803)
0620 04:25:07.559006 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:07.559009 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:07.559204 (+ 195us) server_negotiation.cc:407] Connection header received
0620 04:25:07.559307 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:07.559309 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:07.559348 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:07.559407 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:25:07.559615 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:07.560133 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:07.560963 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:07.561316 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:07.561419 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:07.561647 (+ 228us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:25:07.561721 (+ 74us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:25:07.561786 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:07.561850 (+ 64us) server_negotiation.cc:299] Negotiation successful
0620 04:25:07.561899 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":30}
I20260620 04:25:07.568248 1947 raft_consensus.cc:1261] T 004fe054beb54731a67406bfde665ed5 P fc94a650d15247b7b08bd5c3eeaa41e6 [term 1 FOLLOWER]: Refusing update from remote peer 8af8ecc3c97a4bc9908aa7f03122832c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:25:07.568292 1667 raft_consensus.cc:1261] T 004fe054beb54731a67406bfde665ed5 P 92efaefa517c473cadbf761e21859ff4 [term 1 FOLLOWER]: Refusing update from remote peer 8af8ecc3c97a4bc9908aa7f03122832c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:25:07.568602 2049 consensus_queue.cc:1035] T 004fe054beb54731a67406bfde665ed5 P 8af8ecc3c97a4bc9908aa7f03122832c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc94a650d15247b7b08bd5c3eeaa41e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:25:07.568759 2024 consensus_queue.cc:1035] T 004fe054beb54731a67406bfde665ed5 P 8af8ecc3c97a4bc9908aa7f03122832c [LEADER]: Connected to new peer: Peer: permanent_uuid: "92efaefa517c473cadbf761e21859ff4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:25:07.609659 1509 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50734:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:25:07.609833 1509 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50734:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:25:07.611891 1509 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fb5b3346cacb41ca9314bc02e26d17a7: Sending DeleteTablet for 3 replicas of tablet 004fe054beb54731a67406bfde665ed5
I20260620 04:25:07.612458 1647 tablet_service.cc:1514] Processing DeleteTablet for tablet 004fe054beb54731a67406bfde665ed5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:25:07 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43140
I20260620 04:25:07.612471 1786 tablet_service.cc:1514] Processing DeleteTablet for tablet 004fe054beb54731a67406bfde665ed5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:25:07 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49882
I20260620 04:25:07.612581 1927 tablet_service.cc:1514] Processing DeleteTablet for tablet 004fe054beb54731a67406bfde665ed5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:25:07 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51314
I20260620 04:25:07.612847 2077 tablet_replica.cc:330] T 004fe054beb54731a67406bfde665ed5 P 8af8ecc3c97a4bc9908aa7f03122832c: stopping tablet replica
I20260620 04:25:07.612970 2077 raft_consensus.cc:2229] T 004fe054beb54731a67406bfde665ed5 P 8af8ecc3c97a4bc9908aa7f03122832c [term 1 LEADER]: Raft consensus shutting down.
I20260620 04:25:07.613073 2076 tablet_replica.cc:330] T 004fe054beb54731a67406bfde665ed5 P 92efaefa517c473cadbf761e21859ff4: stopping tablet replica
I20260620 04:25:07.613178 2076 raft_consensus.cc:2229] T 004fe054beb54731a67406bfde665ed5 P 92efaefa517c473cadbf761e21859ff4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:25:07.613279 2076 raft_consensus.cc:2258] T 004fe054beb54731a67406bfde665ed5 P 92efaefa517c473cadbf761e21859ff4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:25:07.613272 2077 raft_consensus.cc:2258] T 004fe054beb54731a67406bfde665ed5 P 8af8ecc3c97a4bc9908aa7f03122832c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:25:07.613483 2078 tablet_replica.cc:330] T 004fe054beb54731a67406bfde665ed5 P fc94a650d15247b7b08bd5c3eeaa41e6: stopping tablet replica
I20260620 04:25:07.613585 2078 raft_consensus.cc:2229] T 004fe054beb54731a67406bfde665ed5 P fc94a650d15247b7b08bd5c3eeaa41e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:25:07.613688 2078 raft_consensus.cc:2258] T 004fe054beb54731a67406bfde665ed5 P fc94a650d15247b7b08bd5c3eeaa41e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:25:07.613978 2076 ts_tablet_manager.cc:1905] T 004fe054beb54731a67406bfde665ed5 P 92efaefa517c473cadbf761e21859ff4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:25:07.614281 2078 ts_tablet_manager.cc:1905] T 004fe054beb54731a67406bfde665ed5 P fc94a650d15247b7b08bd5c3eeaa41e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:25:07.614629 2077 ts_tablet_manager.cc:1905] T 004fe054beb54731a67406bfde665ed5 P 8af8ecc3c97a4bc9908aa7f03122832c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:25:07.616868 2078 ts_tablet_manager.cc:1918] T 004fe054beb54731a67406bfde665ed5 P fc94a650d15247b7b08bd5c3eeaa41e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 04:25:07.616943 2078 log.cc:1196] T 004fe054beb54731a67406bfde665ed5 P fc94a650d15247b7b08bd5c3eeaa41e6: Deleting WAL directory at /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929496330398-392-0/minicluster-data/ts-2/wal/wals/004fe054beb54731a67406bfde665ed5
I20260620 04:25:07.617197 2078 ts_tablet_manager.cc:1939] T 004fe054beb54731a67406bfde665ed5 P fc94a650d15247b7b08bd5c3eeaa41e6: Deleting consensus metadata
I20260620 04:25:07.617178 2076 ts_tablet_manager.cc:1918] T 004fe054beb54731a67406bfde665ed5 P 92efaefa517c473cadbf761e21859ff4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 04:25:07.617254 2076 log.cc:1196] T 004fe054beb54731a67406bfde665ed5 P 92efaefa517c473cadbf761e21859ff4: Deleting WAL directory at /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929496330398-392-0/minicluster-data/ts-0/wal/wals/004fe054beb54731a67406bfde665ed5
I20260620 04:25:07.617475 2076 ts_tablet_manager.cc:1939] T 004fe054beb54731a67406bfde665ed5 P 92efaefa517c473cadbf761e21859ff4: Deleting consensus metadata
I20260620 04:25:07.617640 2077 ts_tablet_manager.cc:1918] T 004fe054beb54731a67406bfde665ed5 P 8af8ecc3c97a4bc9908aa7f03122832c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 04:25:07.617744 2077 log.cc:1196] T 004fe054beb54731a67406bfde665ed5 P 8af8ecc3c97a4bc9908aa7f03122832c: Deleting WAL directory at /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929496330398-392-0/minicluster-data/ts-1/wal/wals/004fe054beb54731a67406bfde665ed5
I20260620 04:25:07.617887 1495 catalog_manager.cc:4841] TS fc94a650d15247b7b08bd5c3eeaa41e6 (127.0.98.3:37745): tablet 004fe054beb54731a67406bfde665ed5 (table test-table [id=23d3f9a8302042739f1535f154a50705]) successfully deleted
I20260620 04:25:07.618095 2077 ts_tablet_manager.cc:1939] T 004fe054beb54731a67406bfde665ed5 P 8af8ecc3c97a4bc9908aa7f03122832c: Deleting consensus metadata
I20260620 04:25:07.618111 1496 catalog_manager.cc:4841] TS 92efaefa517c473cadbf761e21859ff4 (127.0.98.1:45753): tablet 004fe054beb54731a67406bfde665ed5 (table test-table [id=23d3f9a8302042739f1535f154a50705]) successfully deleted
I20260620 04:25:07.618773 1497 catalog_manager.cc:4841] TS 8af8ecc3c97a4bc9908aa7f03122832c (127.0.98.2:32803): tablet 004fe054beb54731a67406bfde665ed5 (table test-table [id=23d3f9a8302042739f1535f154a50705]) successfully deleted
Jun 20 04:25:07 dist-test-slave-nk6d krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929506, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 04:25:07.627862 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:07.618410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43150 (local address 127.0.98.1:45753)
0620 04:25:07.618513 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:07.618518 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:07.618753 (+ 235us) server_negotiation.cc:407] Connection header received
0620 04:25:07.618960 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:07.618964 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:07.619025 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:07.619109 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:07.619731 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:07.620536 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:07.621328 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:07.621542 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:07.621626 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:07.624714 (+ 3088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:07.624723 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:07.624736 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:25:07.624759 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:07.626278 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:07.626680 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:07.626685 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:07.626686 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:07.626760 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:07.626993 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:07.626996 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:07.626998 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:07.627233 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:07.627412 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:07.627597 (+ 185us) server_negotiation.cc:299] Negotiation successful
0620 04:25:07.627717 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
W20260620 04:25:07.628315 1707 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43150
I20260620 04:25:07.637530 2057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:07.629952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50748 (local address 127.0.98.62:37763)
0620 04:25:07.630077 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:07.630082 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:07.630158 (+ 76us) server_negotiation.cc:407] Connection header received
0620 04:25:07.630260 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:07.630262 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:07.630301 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:07.630359 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:07.631066 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:07.631767 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:07.632885 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:07.633098 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:07.633177 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:07.634626 (+ 1449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:07.634629 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:07.634631 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:25:07.634658 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:07.635858 (+ 1200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:07.636379 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:07.636382 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:07.636383 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:07.636469 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:07.636711 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:07.636713 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:07.636714 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:07.636899 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:07.636995 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:07.637251 (+ 256us) server_negotiation.cc:299] Negotiation successful
0620 04:25:07.637378 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260620 04:25:07.637905 1509 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50748
I20260620 04:25:07.639168 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 1582
I20260620 04:25:07.647442 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 1722
I20260620 04:25:07.653149 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 1862
I20260620 04:25:07.657756 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 1478
2026-06-20T04:25:07Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (5410 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929496330398-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:25:07 dist-test-slave-nk6d krb5kdc[2085](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:25:07 dist-test-slave-nk6d krb5kdc[2085](info): set up 2 sockets
Jun 20 04:25:07 dist-test-slave-nk6d krb5kdc[2085](info): commencing operation
krb5kdc: starting...
W20260620 04:25:08.603650 392 mini_kdc.cc:121] Time spent starting KDC: real 0.925s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929496330398-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929496330398-392-0/krb5kdc/test-user.keytab.
Jun 20 04:25:08 dist-test-slave-nk6d krb5kdc[2085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929508, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:25:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:25:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 04:25:08.714784 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929496330398-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929496330398-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929496330398-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34081
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34081
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:25:08.822633 2098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:25:08.822896 2098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:25:08.822950 2098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:25:08.827003 2098 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:25:08.827060 2098 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 04:25:08.827083 2098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:25:08.827105 2098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:25:08.827135 2098 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:25:08.827183 2098 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:25:08.832417 2098 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929496330398-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929496330398-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34081
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34081
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2098
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929496330398-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:25:08.833434 2098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:25:08.834421 2098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:25:08.842521 2103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:25:08.842521 2104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:25:08.842669 2098 server_base.cc:1034] running on GCE node
W20260620 04:25:08.842566 2106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:25:08.843287 2098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:25:08.844435 2098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:25:08.845621 2098 hybrid_clock.cc:648] HybridClock initialized: now 1781929508845604 us; error 33 us; skew 500 ppm
Jun 20 04:25:08 dist-test-slave-nk6d krb5kdc[2085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929508, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:25:08.852480 2098 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 04:25:08.853354 2098 webserver.cc:460] Webserver started at http://127.0.98.62:33721/ using document root <none> and password file <none>
I20260620 04:25:08.853886 2098 fs_manager.cc:359] Metadata directory not provided
I20260620 04:25:08.853945 2098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:25:08.854105 2098 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:25:08.856078 2098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929496330398-392-0/minicluster-data/master-0/data/instance:
uuid: "15ec2623f54b4519af25addf17f1c427"
format_stamp: "Formatted at 2026-06-20 04:25:08 on dist-test-slave-nk6d"
server_key: "f4fdca3e6fb8d65555df8dbbeebffebf"
server_key_iv: "14c7291638c97ab231abc47a7438a745"
server_key_version: "encryptionkey@0"
I20260620 04:25:08.856581 2098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929496330398-392-0/minicluster-data/master-0/wal/instance:
uuid: "15ec2623f54b4519af25addf17f1c427"
format_stamp: "Formatted at 2026-06-20 04:25:08 on dist-test-slave-nk6d"
server_key: "f4fdca3e6fb8d65555df8dbbeebffebf"
server_key_iv: "14c7291638c97ab231abc47a7438a745"
server_key_version: "encryptionkey@0"
I20260620 04:25:08.860143 2098 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:25:08.862427 2113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:08.863703 2098 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:25:08.863816 2098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929496330398-392-0/minicluster-data/master-0/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929496330398-392-0/minicluster-data/master-0/wal
uuid: "15ec2623f54b4519af25addf17f1c427"
format_stamp: "Formatted at 2026-06-20 04:25:08 on dist-test-slave-nk6d"
server_key: "f4fdca3e6fb8d65555df8dbbeebffebf"
server_key_iv: "14c7291638c97ab231abc47a7438a745"
server_key_version: "encryptionkey@0"
I20260620 04:25:08.863909 2098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929496330398-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929496330398-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929496330398-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:25:08.902109 2098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:25:08.905448 2098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:25:08.905646 2098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:25:08.913230 2098 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34081
I20260620 04:25:08.913246 2175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34081 every 8 connection(s)
I20260620 04:25:08.914165 2098 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:25:08.916917 2176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:25:08.920562 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 2098
I20260620 04:25:08.920665 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929496330398-392-0/minicluster-data/master-0/wal/instance
I20260620 04:25:08.920939 392 external_mini_cluster.cc:1428] Setting key ded7e0144592fc7f7ff5a791c495d495
I20260620 04:25:08.922525 2176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15ec2623f54b4519af25addf17f1c427: Bootstrap starting.
I20260620 04:25:08.925194 2176 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 15ec2623f54b4519af25addf17f1c427: Neither blocks nor log segments found. Creating new log.
I20260620 04:25:08.926044 2176 log.cc:824] T 00000000000000000000000000000000 P 15ec2623f54b4519af25addf17f1c427: Log is configured to *not* fsync() on all Append() calls
I20260620 04:25:08.928150 2176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15ec2623f54b4519af25addf17f1c427: No bootstrap required, opened a new log
Jun 20 04:25:08 dist-test-slave-nk6d krb5kdc[2085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929508, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:25:08.931808 2176 raft_consensus.cc:348] T 00000000000000000000000000000000 P 15ec2623f54b4519af25addf17f1c427 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15ec2623f54b4519af25addf17f1c427" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34081 } }
I20260620 04:25:08.932104 2176 raft_consensus.cc:374] T 00000000000000000000000000000000 P 15ec2623f54b4519af25addf17f1c427 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:25:08.932169 2176 raft_consensus.cc:729] T 00000000000000000000000000000000 P 15ec2623f54b4519af25addf17f1c427 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15ec2623f54b4519af25addf17f1c427, State: Initialized, Role: FOLLOWER
I20260620 04:25:08.932667 2176 consensus_queue.cc:260] T 00000000000000000000000000000000 P 15ec2623f54b4519af25addf17f1c427 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15ec2623f54b4519af25addf17f1c427" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34081 } }
I20260620 04:25:08.932801 2176 raft_consensus.cc:388] T 00000000000000000000000000000000 P 15ec2623f54b4519af25addf17f1c427 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:25:08.932858 2176 raft_consensus.cc:482] T 00000000000000000000000000000000 P 15ec2623f54b4519af25addf17f1c427 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:25:08.932951 2176 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 15ec2623f54b4519af25addf17f1c427 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:25:08.933775 2176 raft_consensus.cc:504] T 00000000000000000000000000000000 P 15ec2623f54b4519af25addf17f1c427 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15ec2623f54b4519af25addf17f1c427" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34081 } }
I20260620 04:25:08.934128 2176 leader_election.cc:302] T 00000000000000000000000000000000 P 15ec2623f54b4519af25addf17f1c427 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 15ec2623f54b4519af25addf17f1c427; no voters:
I20260620 04:25:08.934510 2176 leader_election.cc:288] T 00000000000000000000000000000000 P 15ec2623f54b4519af25addf17f1c427 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:25:08.934685 2181 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 15ec2623f54b4519af25addf17f1c427 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:25:08.935003 2181 raft_consensus.cc:686] T 00000000000000000000000000000000 P 15ec2623f54b4519af25addf17f1c427 [term 1 LEADER]: Becoming Leader. State: Replica: 15ec2623f54b4519af25addf17f1c427, State: Running, Role: LEADER
I20260620 04:25:08.935389 2181 consensus_queue.cc:237] T 00000000000000000000000000000000 P 15ec2623f54b4519af25addf17f1c427 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15ec2623f54b4519af25addf17f1c427" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34081 } }
I20260620 04:25:08.935717 2176 sys_catalog.cc:564] T 00000000000000000000000000000000 P 15ec2623f54b4519af25addf17f1c427 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:25:08.935798 2179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:08.922296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52022 (local address 127.0.98.62:34081)
0620 04:25:08.922698 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:08.922720 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:08.922764 (+ 44us) server_negotiation.cc:407] Connection header received
0620 04:25:08.923571 (+ 807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:08.923593 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:08.924075 (+ 482us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:08.924450 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:08.925080 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:08.926524 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:08.927426 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:08.928110 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:08.928272 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:08.930589 (+ 2317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:08.930595 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:08.930613 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:25:08.930643 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:08.932496 (+ 1853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:08.933000 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:08.933006 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:08.933011 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:08.933156 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:08.933441 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:08.933445 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:08.933447 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:08.933901 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:08.934083 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:08.934382 (+ 299us) server_negotiation.cc:299] Negotiation successful
0620 04:25:08.934724 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":111,"threads_started":1}
I20260620 04:25:08.937005 2183 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15ec2623f54b4519af25addf17f1c427 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 15ec2623f54b4519af25addf17f1c427. Latest consensus state: current_term: 1 leader_uuid: "15ec2623f54b4519af25addf17f1c427" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15ec2623f54b4519af25addf17f1c427" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34081 } } }
I20260620 04:25:08.937144 2183 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15ec2623f54b4519af25addf17f1c427 [sys.catalog]: This master's current role is: LEADER
I20260620 04:25:08.937474 2182 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15ec2623f54b4519af25addf17f1c427 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "15ec2623f54b4519af25addf17f1c427" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15ec2623f54b4519af25addf17f1c427" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34081 } } }
I20260620 04:25:08.937565 2182 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15ec2623f54b4519af25addf17f1c427 [sys.catalog]: This master's current role is: LEADER
I20260620 04:25:08.938026 2190 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:25:08.941181 2190 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:25:08.946278 2190 catalog_manager.cc:1347] Generated new cluster ID: 623b634db6e74df7aa2be497ad86c7a5
I20260620 04:25:08.946362 2190 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:25:08.959090 2190 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:25:08.960433 2190 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:25:08.975549 2190 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 15ec2623f54b4519af25addf17f1c427: Generated new TSK 0
I20260620 04:25:08.976320 2190 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929496330398-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929496330398-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 04:25:09.003269 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929496330398-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929496330398-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929496330398-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929496330398-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34081
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929496330398-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:25:09.111697 2202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:25:09.111986 2202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:25:09.112037 2202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:25:09.116009 2202 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:25:09.116070 2202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:25:09.116109 2202 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 04:25:09.116122 2202 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 04:25:09.116154 2202 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 04:25:09.121356 2202 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929496330398-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929496330398-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929496330398-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929496330398-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:34081
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2202
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929496330398-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:25:09.122376 2202 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:25:09.123364 2202 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 04:25:09.132578 2202 server_base.cc:1034] running on GCE node
W20260620 04:25:09.132582 2207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:25:09.132524 2208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:25:09.132737 2210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:25:09.133123 2202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:25:09.133736 2202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:25:09.134907 2202 hybrid_clock.cc:648] HybridClock initialized: now 1781929509134885 us; error 32 us; skew 500 ppm
Jun 20 04:25:09 dist-test-slave-nk6d krb5kdc[2085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929509, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:25:09.137761 2202 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 04:25:09.138625 2202 webserver.cc:460] Webserver started at http://127.0.98.1:44575/ using document root <none> and password file <none>
I20260620 04:25:09.139171 2202 fs_manager.cc:359] Metadata directory not provided
I20260620 04:25:09.139233 2202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:25:09.139433 2202 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:25:09.141495 2202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929496330398-392-0/minicluster-data/ts-0/data/instance:
uuid: "3d79f1af503743878943a0d3ac3fe630"
format_stamp: "Formatted at 2026-06-20 04:25:09 on dist-test-slave-nk6d"
server_key: "7bbbe6c2dbee847dfca3030464665fb9"
server_key_iv: "2e95fe66f986836b7c92a087523b2ef1"
server_key_version: "encryptionkey@0"
I20260620 04:25:09.142025 2202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929496330398-392-0/minicluster-data/ts-0/wal/instance:
uuid: "3d79f1af503743878943a0d3ac3fe630"
format_stamp: "Formatted at 2026-06-20 04:25:09 on dist-test-slave-nk6d"
server_key: "7bbbe6c2dbee847dfca3030464665fb9"
server_key_iv: "2e95fe66f986836b7c92a087523b2ef1"
server_key_version: "encryptionkey@0"
I20260620 04:25:09.145731 2202 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260620 04:25:09.148142 2217 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:09.149391 2202 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:25:09.149497 2202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929496330398-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929496330398-392-0/minicluster-data/ts-0/wal
uuid: "3d79f1af503743878943a0d3ac3fe630"
format_stamp: "Formatted at 2026-06-20 04:25:09 on dist-test-slave-nk6d"
server_key: "7bbbe6c2dbee847dfca3030464665fb9"
server_key_iv: "2e95fe66f986836b7c92a087523b2ef1"
server_key_version: "encryptionkey@0"
I20260620 04:25:09.149595 2202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929496330398-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929496330398-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929496330398-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:25:09.171720 2202 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:25:09.174702 2202 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:25:09.174880 2202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:25:09.176326 2202 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:25:09.176374 2202 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:09.176426 2202 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:25:09.176450 2202 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 04:25:09 dist-test-slave-nk6d krb5kdc[2085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929509, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:25:09.186340 2202 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37973
I20260620 04:25:09.186518 2332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37973 every 8 connection(s)
I20260620 04:25:09.187384 2202 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929496330398-392-0/minicluster-data/ts-0/data/info.pb
I20260620 04:25:09.189251 2179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:09.177260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60375 (local address 127.0.98.62:34081)
0620 04:25:09.177367 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:09.177371 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:09.178527 (+ 1156us) server_negotiation.cc:407] Connection header received
0620 04:25:09.179553 (+ 1026us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:09.179557 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:09.179613 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:09.179690 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:09.180938 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:09.181458 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:09.182682 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:09.183045 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:09.183164 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:09.185564 (+ 2400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:09.185568 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:09.185570 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:25:09.185598 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:09.187244 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:09.187754 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:09.187758 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:09.187760 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:09.187857 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:09.188275 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:09.188279 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:09.188281 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:09.188478 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:09.188555 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:09.189005 (+ 450us) server_negotiation.cc:299] Negotiation successful
0620 04:25:09.189107 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 04:25:09.189637 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 2202
I20260620 04:25:09.189738 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929496330398-392-0/minicluster-data/ts-0/wal/instance
I20260620 04:25:09.189985 392 external_mini_cluster.cc:1428] Setting key 5191cce8f1c4ae57d689292e4e4c7593
I20260620 04:25:09.189986 2226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:09.177850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34081 (local address 127.0.98.1:60375)
0620 04:25:09.178364 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:09.178411 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:09.179330 (+ 919us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:09.179891 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:09.179897 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:09.180297 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:09.180802 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:09.180816 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:09.181602 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:09.181606 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:09.182552 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:09.182568 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:09.183207 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:09.183212 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:09.183400 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:09.183950 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:09.183995 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:09.185409 (+ 1414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:25:09.187358 (+ 1949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:09.187363 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:09.187380 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:09.187636 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:09.187940 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:09.187943 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:09.187945 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:09.188120 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:09.188571 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:25:09.188576 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:25:09.188788 (+ 212us) client_negotiation.cc:769] Sending connection context
0620 04:25:09.188937 (+ 149us) client_negotiation.cc:240] Negotiation successful
0620 04:25:09.189176 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":131,"threads_started":1}
I20260620 04:25:09.191260 2333 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34081
I20260620 04:25:09.191434 2333 heartbeater.cc:464] Registering TS with master...
I20260620 04:25:09.191920 2333 heartbeater.cc:511] Master 127.0.98.62:34081 requested a full tablet report, sending...
I20260620 04:25:09.193434 2129 ts_manager.cc:194] Registered new tserver with Master: 3d79f1af503743878943a0d3ac3fe630 (127.0.98.1:37973)
I20260620 04:25:09.194885 2129 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60375
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 04:25:09.205643 2340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:09.193253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34081 (local address 127.0.98.1:48589)
0620 04:25:09.198965 (+ 5712us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:09.198979 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:09.199081 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:09.199295 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:09.199296 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:09.199500 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:09.199600 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:09.199606 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:09.200450 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:09.200452 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:09.201104 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:09.201114 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:09.201546 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:09.201550 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:09.201640 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:09.202141 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:09.202155 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:09.202554 (+ 399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:25:09.204290 (+ 1736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:09.204293 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:09.204294 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:09.204484 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:09.204756 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:09.204759 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:09.204760 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:09.204862 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:09.205242 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:25:09.205244 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:25:09.205312 (+ 68us) client_negotiation.cc:769] Sending connection context
0620 04:25:09.205391 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 04:25:09.205491 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5628,"thread_start_us":100,"threads_started":1}
I20260620 04:25:09.205746 2179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:09.193302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48589 (local address 127.0.98.62:34081)
0620 04:25:09.193410 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:09.193414 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:09.199074 (+ 5660us) server_negotiation.cc:407] Connection header received
0620 04:25:09.199146 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:09.199148 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:09.199195 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:09.199263 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:09.199710 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:09.200286 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:09.201223 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:09.201438 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:09.201530 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:09.202697 (+ 1167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:09.202701 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:09.202703 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:25:09.202730 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:09.204181 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:09.204569 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:09.204573 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:09.204575 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:09.204664 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:09.204941 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:09.204944 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:09.204946 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:09.205155 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:09.205261 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:09.205464 (+ 203us) server_negotiation.cc:299] Negotiation successful
0620 04:25:09.205583 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929496330398-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929496330398-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 04:25:09.215559 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929496330398-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929496330398-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929496330398-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929496330398-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34081
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929496330398-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:25:09.322108 2342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:25:09.322378 2342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:25:09.322441 2342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:25:09.326416 2342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:25:09.326475 2342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:25:09.326527 2342 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 04:25:09.326552 2342 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 04:25:09.326606 2342 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 04:25:09.331919 2342 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929496330398-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929496330398-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929496330398-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929496330398-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:34081
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2342
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929496330398-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:25:09.332949 2342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:25:09.333932 2342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:25:09.343043 2347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:25:09.343060 2348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:25:09.343065 2350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:25:09.343391 2342 server_base.cc:1034] running on GCE node
I20260620 04:25:09.343807 2342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:25:09.344467 2342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:25:09.345614 2342 hybrid_clock.cc:648] HybridClock initialized: now 1781929509345587 us; error 39 us; skew 500 ppm
Jun 20 04:25:09 dist-test-slave-nk6d krb5kdc[2085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929509, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:25:09.348168 2342 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 04:25:09.348963 2342 webserver.cc:460] Webserver started at http://127.0.98.2:44989/ using document root <none> and password file <none>
I20260620 04:25:09.349483 2342 fs_manager.cc:359] Metadata directory not provided
I20260620 04:25:09.349546 2342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:25:09.349725 2342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:25:09.351728 2342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929496330398-392-0/minicluster-data/ts-1/data/instance:
uuid: "d4cfed1c9d714d02a3797ec591c46bed"
format_stamp: "Formatted at 2026-06-20 04:25:09 on dist-test-slave-nk6d"
server_key: "ddf28871cc103295c4a26c38efba6640"
server_key_iv: "33b3e73e25337414fa00463c68d3a352"
server_key_version: "encryptionkey@0"
I20260620 04:25:09.352306 2342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929496330398-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d4cfed1c9d714d02a3797ec591c46bed"
format_stamp: "Formatted at 2026-06-20 04:25:09 on dist-test-slave-nk6d"
server_key: "ddf28871cc103295c4a26c38efba6640"
server_key_iv: "33b3e73e25337414fa00463c68d3a352"
server_key_version: "encryptionkey@0"
I20260620 04:25:09.355818 2342 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:25:09.358184 2357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:09.359370 2342 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:25:09.359472 2342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929496330398-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929496330398-392-0/minicluster-data/ts-1/wal
uuid: "d4cfed1c9d714d02a3797ec591c46bed"
format_stamp: "Formatted at 2026-06-20 04:25:09 on dist-test-slave-nk6d"
server_key: "ddf28871cc103295c4a26c38efba6640"
server_key_iv: "33b3e73e25337414fa00463c68d3a352"
server_key_version: "encryptionkey@0"
I20260620 04:25:09.359557 2342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929496330398-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929496330398-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929496330398-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:25:09.388608 2342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:25:09.391777 2342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:25:09.391950 2342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:25:09.393401 2342 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:25:09.393448 2342 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:09.393500 2342 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:25:09.393528 2342 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 04:25:09 dist-test-slave-nk6d krb5kdc[2085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929509, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:25:09.403036 2342 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38267
I20260620 04:25:09.403074 2472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38267 every 8 connection(s)
I20260620 04:25:09.403976 2342 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929496330398-392-0/minicluster-data/ts-1/data/info.pb
I20260620 04:25:09.405843 2179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:09.394265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57401 (local address 127.0.98.62:34081)
0620 04:25:09.394404 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:09.394409 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:09.395269 (+ 860us) server_negotiation.cc:407] Connection header received
0620 04:25:09.396390 (+ 1121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:09.396394 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:09.396447 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:09.396520 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:09.397729 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:09.398311 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:09.399489 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:09.399758 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:09.399845 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:09.402543 (+ 2698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:09.402546 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:09.402548 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:25:09.402572 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:09.403998 (+ 1426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:09.404517 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:09.404521 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:09.404523 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:09.404607 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:09.404949 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:09.404952 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:09.404953 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:09.405116 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:09.405185 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:09.405623 (+ 438us) server_negotiation.cc:299] Negotiation successful
0620 04:25:09.405727 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 04:25:09.406592 2366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:09.394677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34081 (local address 127.0.98.2:57401)
0620 04:25:09.395109 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:09.395157 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:09.396176 (+ 1019us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:09.396710 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:09.396717 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:09.397090 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:09.397595 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:09.397608 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:09.398424 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:09.398428 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:09.399383 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:09.399391 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:09.399918 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:09.399922 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:09.400124 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:09.400648 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:09.400663 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:09.402385 (+ 1722us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:25:09.404121 (+ 1736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:09.404125 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:09.404136 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:09.404397 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:09.404690 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:09.404694 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:09.404695 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:09.404861 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:09.405190 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:25:09.405195 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:25:09.405429 (+ 234us) client_negotiation.cc:769] Sending connection context
0620 04:25:09.405587 (+ 158us) client_negotiation.cc:240] Negotiation successful
0620 04:25:09.405823 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":101,"threads_started":1}
I20260620 04:25:09.407703 2473 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34081
I20260620 04:25:09.407842 2473 heartbeater.cc:464] Registering TS with master...
I20260620 04:25:09.408315 2473 heartbeater.cc:511] Master 127.0.98.62:34081 requested a full tablet report, sending...
I20260620 04:25:09.409246 2128 ts_manager.cc:194] Registered new tserver with Master: d4cfed1c9d714d02a3797ec591c46bed (127.0.98.2:38267)
I20260620 04:25:09.409842 2128 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:57401
I20260620 04:25:09.411268 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 2342
I20260620 04:25:09.411373 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929496330398-392-0/minicluster-data/ts-1/wal/instance
I20260620 04:25:09.411646 392 external_mini_cluster.cc:1428] Setting key f7d8a25be63a18bfee884612c5904c6a
I20260620 04:25:09.417213 2179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:09.409673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49831 (local address 127.0.98.62:34081)
0620 04:25:09.409784 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:09.409788 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:09.409986 (+ 198us) server_negotiation.cc:407] Connection header received
0620 04:25:09.410087 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:09.410090 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:09.410129 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:09.410188 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:09.410572 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:09.411074 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:09.411938 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:09.412189 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:09.412292 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:09.414559 (+ 2267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:09.414563 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:09.414565 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:25:09.414588 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:09.415740 (+ 1152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:09.416147 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:09.416151 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:09.416153 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:09.416230 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:09.416456 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:09.416460 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:09.416461 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:09.416651 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:09.416733 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:09.416963 (+ 230us) server_negotiation.cc:299] Negotiation successful
0620 04:25:09.417059 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 04:25:09.417217 2479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:09.409610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34081 (local address 127.0.98.2:49831)
0620 04:25:09.409902 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:09.409915 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:09.410005 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:09.410211 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:09.410214 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:09.410390 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:09.410475 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:09.410480 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:09.411185 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:09.411188 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:09.411832 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:09.411841 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:09.412295 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:09.412298 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:09.412374 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:09.414128 (+ 1754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:09.414139 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:09.414438 (+ 299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:25:09.415834 (+ 1396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:09.415837 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:09.415839 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:09.416069 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:09.416306 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:09.416309 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:09.416311 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:09.416384 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:09.416741 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:25:09.416744 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:25:09.416826 (+ 82us) client_negotiation.cc:769] Sending connection context
0620 04:25:09.416906 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 04:25:09.417036 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":65,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929496330398-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929496330398-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 04:25:09.434964 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929496330398-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929496330398-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929496330398-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929496330398-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34081
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929496330398-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:25:09.541563 2482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:25:09.541885 2482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:25:09.541957 2482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:25:09.547678 2482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:25:09.547760 2482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:25:09.547820 2482 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 04:25:09.547839 2482 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 04:25:09.547886 2482 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 04:25:09.555063 2482 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929496330398-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929496330398-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929496330398-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929496330398-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:34081
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2482
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929496330398-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:25:09.556191 2482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:25:09.557219 2482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:25:09.566596 2487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:25:09.566604 2488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:25:09.566888 2482 server_base.cc:1034] running on GCE node
W20260620 04:25:09.567106 2490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:25:09.567528 2482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:25:09.568246 2482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:25:09.569422 2482 hybrid_clock.cc:648] HybridClock initialized: now 1781929509569385 us; error 52 us; skew 500 ppm
Jun 20 04:25:09 dist-test-slave-nk6d krb5kdc[2085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929509, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:25:09.572180 2482 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 04:25:09.573005 2482 webserver.cc:460] Webserver started at http://127.0.98.3:38279/ using document root <none> and password file <none>
I20260620 04:25:09.573513 2482 fs_manager.cc:359] Metadata directory not provided
I20260620 04:25:09.573567 2482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:25:09.573743 2482 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:25:09.575667 2482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929496330398-392-0/minicluster-data/ts-2/data/instance:
uuid: "d97a29850cc548d18b882742057e11dd"
format_stamp: "Formatted at 2026-06-20 04:25:09 on dist-test-slave-nk6d"
server_key: "9cb28c4d9ac72dd51ed8989c411dad1b"
server_key_iv: "5d5eae0fcc4c19d2cf0f41ef43ec5e7b"
server_key_version: "encryptionkey@0"
I20260620 04:25:09.576193 2482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929496330398-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d97a29850cc548d18b882742057e11dd"
format_stamp: "Formatted at 2026-06-20 04:25:09 on dist-test-slave-nk6d"
server_key: "9cb28c4d9ac72dd51ed8989c411dad1b"
server_key_iv: "5d5eae0fcc4c19d2cf0f41ef43ec5e7b"
server_key_version: "encryptionkey@0"
I20260620 04:25:09.579697 2482 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260620 04:25:09.582062 2497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:09.583310 2482 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 04:25:09.583410 2482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929496330398-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929496330398-392-0/minicluster-data/ts-2/wal
uuid: "d97a29850cc548d18b882742057e11dd"
format_stamp: "Formatted at 2026-06-20 04:25:09 on dist-test-slave-nk6d"
server_key: "9cb28c4d9ac72dd51ed8989c411dad1b"
server_key_iv: "5d5eae0fcc4c19d2cf0f41ef43ec5e7b"
server_key_version: "encryptionkey@0"
I20260620 04:25:09.583503 2482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929496330398-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929496330398-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929496330398-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:25:09.596660 2482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:25:09.599699 2482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:25:09.599869 2482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:25:09.601238 2482 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:25:09.601280 2482 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:09.601320 2482 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:25:09.601348 2482 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 04:25:09 dist-test-slave-nk6d krb5kdc[2085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929509, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:25:09.610814 2482 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45271
I20260620 04:25:09.610857 2612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45271 every 8 connection(s)
I20260620 04:25:09.611827 2482 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929496330398-392-0/minicluster-data/ts-2/data/info.pb
I20260620 04:25:09.614367 2179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:09.602117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48521 (local address 127.0.98.62:34081)
0620 04:25:09.602224 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:09.602228 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:09.603261 (+ 1033us) server_negotiation.cc:407] Connection header received
0620 04:25:09.604316 (+ 1055us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:09.604320 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:09.604376 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:09.604470 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:09.605831 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:09.606619 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:09.608058 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:09.608375 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:09.608477 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:09.611002 (+ 2525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:09.611006 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:09.611009 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:25:09.611037 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:09.612622 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:09.613102 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:09.613105 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:09.613106 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:09.613187 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:09.613451 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:09.613453 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:09.613454 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:09.613617 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:09.613685 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:09.614124 (+ 439us) server_negotiation.cc:299] Negotiation successful
0620 04:25:09.614247 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 04:25:09.615043 2506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:09.602633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34081 (local address 127.0.98.3:48521)
0620 04:25:09.603100 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:09.603139 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:09.604099 (+ 960us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:09.604653 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:09.604660 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:09.605056 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:09.605671 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:09.605690 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:09.606753 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:09.606756 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:09.607911 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:09.607923 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:09.608511 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:09.608515 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:09.608685 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:09.609275 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:09.609295 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:09.610811 (+ 1516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:25:09.612722 (+ 1911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:09.612728 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:09.612742 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:09.613019 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:09.613264 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:09.613266 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:09.613267 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:09.613385 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:09.613683 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:25:09.613688 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:25:09.613897 (+ 209us) client_negotiation.cc:769] Sending connection context
0620 04:25:09.614039 (+ 142us) client_negotiation.cc:240] Negotiation successful
0620 04:25:09.614277 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":128,"threads_started":1}
I20260620 04:25:09.616196 2613 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34081
I20260620 04:25:09.616339 2613 heartbeater.cc:464] Registering TS with master...
I20260620 04:25:09.616889 2613 heartbeater.cc:511] Master 127.0.98.62:34081 requested a full tablet report, sending...
I20260620 04:25:09.618016 2129 ts_manager.cc:194] Registered new tserver with Master: d97a29850cc548d18b882742057e11dd (127.0.98.3:45271)
I20260620 04:25:09.618633 2129 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:48521
I20260620 04:25:09.620723 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 2482
I20260620 04:25:09.620818 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929496330398-392-0/minicluster-data/ts-2/wal/instance
I20260620 04:25:09.621117 392 external_mini_cluster.cc:1428] Setting key b698a667b0ed07ff34f2b2b66b378731
I20260620 04:25:09.622941 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 04:25:09.625442 2619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:09.618366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34081 (local address 127.0.98.3:52177)
0620 04:25:09.618709 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:09.618721 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:09.618843 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:09.619079 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:09.619081 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:09.619283 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:09.619423 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:09.619432 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:09.620227 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:09.620230 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:09.620958 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:09.620967 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:09.621491 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:09.621495 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:09.621584 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:09.622044 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:09.622057 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:09.622370 (+ 313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:25:09.624058 (+ 1688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:09.624062 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:09.624063 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:09.624296 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:09.624575 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:09.624578 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:09.624580 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:09.624672 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:09.625059 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:25:09.625061 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:25:09.625127 (+ 66us) client_negotiation.cc:769] Sending connection context
0620 04:25:09.625197 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 04:25:09.625294 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":99,"threads_started":1}
I20260620 04:25:09.625502 2179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:09.618457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52177 (local address 127.0.98.62:34081)
0620 04:25:09.618579 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:09.618584 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:09.618779 (+ 195us) server_negotiation.cc:407] Connection header received
0620 04:25:09.618937 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:09.618940 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:09.618983 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:09.619063 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:09.619555 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:09.620098 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:09.621061 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:09.621346 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:09.621458 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:09.622479 (+ 1021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:09.622482 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:09.622485 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:25:09.622509 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:09.623929 (+ 1420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:09.624390 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:09.624393 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:09.624394 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:09.624488 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:09.624745 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:09.624748 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:09.624749 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:09.624984 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:09.625074 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:09.625255 (+ 181us) server_negotiation.cc:299] Negotiation successful
0620 04:25:09.625355 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
Jun 20 04:25:09 dist-test-slave-nk6d krb5kdc[2085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929509, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 04:25:09 dist-test-slave-nk6d krb5kdc[2085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929509, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:25:09.646147 2179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:09.638932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52038 (local address 127.0.98.62:34081)
0620 04:25:09.639055 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:09.639058 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:09.639071 (+ 13us) server_negotiation.cc:407] Connection header received
0620 04:25:09.639107 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:09.639109 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:09.639147 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:09.639214 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:09.639830 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:09.640409 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:09.641167 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:09.641378 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:09.641448 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:09.643477 (+ 2029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:09.643480 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:09.643482 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:25:09.643504 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:09.644748 (+ 1244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:09.645107 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:09.645111 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:09.645113 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:09.645180 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:09.645415 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:09.645419 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:09.645420 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:09.645575 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:09.645683 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:09.645896 (+ 213us) server_negotiation.cc:299] Negotiation successful
0620 04:25:09.646002 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 04:25:09.648624 2129 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52038:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 04:25:09.651502 2129 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 04:25:09.665704 2632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:09.659722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34240 (local address 127.0.98.2:38267)
0620 04:25:09.661254 (+ 1532us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:09.661265 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:09.661286 (+ 21us) server_negotiation.cc:407] Connection header received
0620 04:25:09.661356 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:09.661360 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:09.661547 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:09.661885 (+ 338us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:25:09.661984 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:09.662733 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:09.664533 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:09.665064 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:09.665157 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:09.665315 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:09.665466 (+ 151us) server_negotiation.cc:299] Negotiation successful
0620 04:25:09.665555 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1435,"thread_start_us":107,"threads_started":1}
I20260620 04:25:09.665695 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:09.659648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37973 (local address 127.0.0.1:48964)
0620 04:25:09.660516 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:09.660564 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:09.660711 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:09.661316 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:09.661322 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:09.661343 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:25:09.661470 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:09.661476 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:09.663047 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:09.663050 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:09.664209 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:09.664218 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:09.665302 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:09.665306 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:09.665386 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:09.665407 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 04:25:09.665456 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 04:25:09.665563 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":795,"mutex_wait_us":44,"spinlock_wait_cycles":3712,"thread_start_us":97,"threads_started":1}
I20260620 04:25:09.665881 2635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:09.660326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45271 (local address 127.0.0.1:34728)
0620 04:25:09.661025 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:09.661037 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:09.661122 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:09.661606 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:09.661609 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:09.661624 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:25:09.661718 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:09.661724 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:09.663001 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:09.663004 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:09.664360 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:09.664371 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:09.665515 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:09.665519 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:09.665602 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:09.665620 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 04:25:09.665680 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 04:25:09.665747 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"mutex_wait_us":560,"thread_start_us":109,"threads_started":1}
I20260620 04:25:09.666013 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:09.659601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38267 (local address 127.0.0.1:34240)
0620 04:25:09.660517 (+ 916us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:09.660564 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:09.660711 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:09.661683 (+ 972us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:09.661686 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:09.661699 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:25:09.661768 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:09.661773 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:09.662837 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:09.662841 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:09.664409 (+ 1568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:09.664416 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:09.665169 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:09.665173 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:09.665283 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:09.665321 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 04:25:09.665428 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 04:25:09.665866 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":726,"mutex_wait_us":115,"spinlock_wait_cycles":32896,"thread_start_us":67,"threads_started":1}
I20260620 04:25:09.666165 2633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:09.659886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48964 (local address 127.0.98.1:37973)
0620 04:25:09.660456 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:09.660472 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:09.660642 (+ 170us) server_negotiation.cc:407] Connection header received
0620 04:25:09.660861 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:09.660867 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:09.661132 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:09.661617 (+ 485us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:25:09.661731 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:09.662927 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:09.664358 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:09.665166 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:09.665555 (+ 389us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:09.665701 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:09.665758 (+ 57us) server_negotiation.cc:299] Negotiation successful
0620 04:25:09.666030 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":468,"thread_start_us":249,"threads_started":1}
I20260620 04:25:09.666312 2634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:09.660534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34728 (local address 127.0.98.3:45271)
0620 04:25:09.661047 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:09.661058 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:09.661217 (+ 159us) server_negotiation.cc:407] Connection header received
0620 04:25:09.661294 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:09.661298 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:09.661476 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:09.661617 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:25:09.661803 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:09.662900 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:09.664507 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:09.665212 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:09.665313 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:09.665461 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:09.666093 (+ 632us) server_negotiation.cc:299] Negotiation successful
0620 04:25:09.666176 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":106,"threads_started":1}
I20260620 04:25:09.669101 2407 tablet_service.cc:1467] Processing CreateTablet for tablet c8d3334666464f27855f4ef3aa9f43a5 (DEFAULT_TABLE table=test-table [id=303b13f341704793aecca47d268cd87e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:25:09.669101 2267 tablet_service.cc:1467] Processing CreateTablet for tablet c8d3334666464f27855f4ef3aa9f43a5 (DEFAULT_TABLE table=test-table [id=303b13f341704793aecca47d268cd87e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:25:09.670194 2267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8d3334666464f27855f4ef3aa9f43a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:25:09.670194 2407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8d3334666464f27855f4ef3aa9f43a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:25:09.676924 2547 tablet_service.cc:1467] Processing CreateTablet for tablet c8d3334666464f27855f4ef3aa9f43a5 (DEFAULT_TABLE table=test-table [id=303b13f341704793aecca47d268cd87e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:25:09.677867 2547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8d3334666464f27855f4ef3aa9f43a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:25:09.678234 2636 tablet_bootstrap.cc:492] T c8d3334666464f27855f4ef3aa9f43a5 P 3d79f1af503743878943a0d3ac3fe630: Bootstrap starting.
I20260620 04:25:09.678371 2637 tablet_bootstrap.cc:492] T c8d3334666464f27855f4ef3aa9f43a5 P d4cfed1c9d714d02a3797ec591c46bed: Bootstrap starting.
I20260620 04:25:09.680663 2636 tablet_bootstrap.cc:654] T c8d3334666464f27855f4ef3aa9f43a5 P 3d79f1af503743878943a0d3ac3fe630: Neither blocks nor log segments found. Creating new log.
I20260620 04:25:09.681674 2637 tablet_bootstrap.cc:654] T c8d3334666464f27855f4ef3aa9f43a5 P d4cfed1c9d714d02a3797ec591c46bed: Neither blocks nor log segments found. Creating new log.
I20260620 04:25:09.681906 2636 log.cc:824] T c8d3334666464f27855f4ef3aa9f43a5 P 3d79f1af503743878943a0d3ac3fe630: Log is configured to *not* fsync() on all Append() calls
I20260620 04:25:09.682329 2637 log.cc:824] T c8d3334666464f27855f4ef3aa9f43a5 P d4cfed1c9d714d02a3797ec591c46bed: Log is configured to *not* fsync() on all Append() calls
I20260620 04:25:09.683820 2637 tablet_bootstrap.cc:492] T c8d3334666464f27855f4ef3aa9f43a5 P d4cfed1c9d714d02a3797ec591c46bed: No bootstrap required, opened a new log
I20260620 04:25:09.683941 2636 tablet_bootstrap.cc:492] T c8d3334666464f27855f4ef3aa9f43a5 P 3d79f1af503743878943a0d3ac3fe630: No bootstrap required, opened a new log
I20260620 04:25:09.684023 2637 ts_tablet_manager.cc:1397] T c8d3334666464f27855f4ef3aa9f43a5 P d4cfed1c9d714d02a3797ec591c46bed: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 04:25:09.684142 2636 ts_tablet_manager.cc:1397] T c8d3334666464f27855f4ef3aa9f43a5 P 3d79f1af503743878943a0d3ac3fe630: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20260620 04:25:09.685235 2640 tablet_bootstrap.cc:492] T c8d3334666464f27855f4ef3aa9f43a5 P d97a29850cc548d18b882742057e11dd: Bootstrap starting.
I20260620 04:25:09.687302 2640 tablet_bootstrap.cc:654] T c8d3334666464f27855f4ef3aa9f43a5 P d97a29850cc548d18b882742057e11dd: Neither blocks nor log segments found. Creating new log.
I20260620 04:25:09.687404 2636 raft_consensus.cc:348] T c8d3334666464f27855f4ef3aa9f43a5 P 3d79f1af503743878943a0d3ac3fe630 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4cfed1c9d714d02a3797ec591c46bed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38267 } } peers { permanent_uuid: "3d79f1af503743878943a0d3ac3fe630" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "d97a29850cc548d18b882742057e11dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45271 } }
I20260620 04:25:09.687475 2637 raft_consensus.cc:348] T c8d3334666464f27855f4ef3aa9f43a5 P d4cfed1c9d714d02a3797ec591c46bed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4cfed1c9d714d02a3797ec591c46bed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38267 } } peers { permanent_uuid: "3d79f1af503743878943a0d3ac3fe630" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "d97a29850cc548d18b882742057e11dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45271 } }
I20260620 04:25:09.687709 2637 raft_consensus.cc:374] T c8d3334666464f27855f4ef3aa9f43a5 P d4cfed1c9d714d02a3797ec591c46bed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:25:09.687752 2637 raft_consensus.cc:729] T c8d3334666464f27855f4ef3aa9f43a5 P d4cfed1c9d714d02a3797ec591c46bed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4cfed1c9d714d02a3797ec591c46bed, State: Initialized, Role: FOLLOWER
I20260620 04:25:09.687873 2636 raft_consensus.cc:374] T c8d3334666464f27855f4ef3aa9f43a5 P 3d79f1af503743878943a0d3ac3fe630 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:25:09.687947 2636 raft_consensus.cc:729] T c8d3334666464f27855f4ef3aa9f43a5 P 3d79f1af503743878943a0d3ac3fe630 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d79f1af503743878943a0d3ac3fe630, State: Initialized, Role: FOLLOWER
I20260620 04:25:09.688207 2637 consensus_queue.cc:260] T c8d3334666464f27855f4ef3aa9f43a5 P d4cfed1c9d714d02a3797ec591c46bed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4cfed1c9d714d02a3797ec591c46bed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38267 } } peers { permanent_uuid: "3d79f1af503743878943a0d3ac3fe630" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "d97a29850cc548d18b882742057e11dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45271 } }
I20260620 04:25:09.688360 2636 consensus_queue.cc:260] T c8d3334666464f27855f4ef3aa9f43a5 P 3d79f1af503743878943a0d3ac3fe630 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4cfed1c9d714d02a3797ec591c46bed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38267 } } peers { permanent_uuid: "3d79f1af503743878943a0d3ac3fe630" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "d97a29850cc548d18b882742057e11dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45271 } }
I20260620 04:25:09.688483 2640 log.cc:824] T c8d3334666464f27855f4ef3aa9f43a5 P d97a29850cc548d18b882742057e11dd: Log is configured to *not* fsync() on all Append() calls
I20260620 04:25:09.688999 2333 heartbeater.cc:503] Master 127.0.98.62:34081 was elected leader, sending a full tablet report...
I20260620 04:25:09.689170 2637 ts_tablet_manager.cc:1428] T c8d3334666464f27855f4ef3aa9f43a5 P d4cfed1c9d714d02a3797ec591c46bed: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 04:25:09.689375 2636 ts_tablet_manager.cc:1428] T c8d3334666464f27855f4ef3aa9f43a5 P 3d79f1af503743878943a0d3ac3fe630: Time spent starting tablet: real 0.005s user 0.000s sys 0.005s
I20260620 04:25:09.689970 2473 heartbeater.cc:503] Master 127.0.98.62:34081 was elected leader, sending a full tablet report...
I20260620 04:25:09.690995 2640 tablet_bootstrap.cc:492] T c8d3334666464f27855f4ef3aa9f43a5 P d97a29850cc548d18b882742057e11dd: No bootstrap required, opened a new log
I20260620 04:25:09.691164 2640 ts_tablet_manager.cc:1397] T c8d3334666464f27855f4ef3aa9f43a5 P d97a29850cc548d18b882742057e11dd: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260620 04:25:09.694168 2640 raft_consensus.cc:348] T c8d3334666464f27855f4ef3aa9f43a5 P d97a29850cc548d18b882742057e11dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4cfed1c9d714d02a3797ec591c46bed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38267 } } peers { permanent_uuid: "3d79f1af503743878943a0d3ac3fe630" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "d97a29850cc548d18b882742057e11dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45271 } }
I20260620 04:25:09.694366 2640 raft_consensus.cc:374] T c8d3334666464f27855f4ef3aa9f43a5 P d97a29850cc548d18b882742057e11dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:25:09.694399 2640 raft_consensus.cc:729] T c8d3334666464f27855f4ef3aa9f43a5 P d97a29850cc548d18b882742057e11dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d97a29850cc548d18b882742057e11dd, State: Initialized, Role: FOLLOWER
I20260620 04:25:09.694756 2640 consensus_queue.cc:260] T c8d3334666464f27855f4ef3aa9f43a5 P d97a29850cc548d18b882742057e11dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4cfed1c9d714d02a3797ec591c46bed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38267 } } peers { permanent_uuid: "3d79f1af503743878943a0d3ac3fe630" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "d97a29850cc548d18b882742057e11dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45271 } }
I20260620 04:25:09.695372 2613 heartbeater.cc:503] Master 127.0.98.62:34081 was elected leader, sending a full tablet report...
I20260620 04:25:09.695540 2640 ts_tablet_manager.cc:1428] T c8d3334666464f27855f4ef3aa9f43a5 P d97a29850cc548d18b882742057e11dd: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260620 04:25:09.823639 2641 raft_consensus.cc:482] T c8d3334666464f27855f4ef3aa9f43a5 P d4cfed1c9d714d02a3797ec591c46bed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:25:09.823843 2641 raft_consensus.cc:504] T c8d3334666464f27855f4ef3aa9f43a5 P d4cfed1c9d714d02a3797ec591c46bed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4cfed1c9d714d02a3797ec591c46bed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38267 } } peers { permanent_uuid: "3d79f1af503743878943a0d3ac3fe630" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "d97a29850cc548d18b882742057e11dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45271 } }
I20260620 04:25:09.825037 2641 leader_election.cc:288] T c8d3334666464f27855f4ef3aa9f43a5 P d4cfed1c9d714d02a3797ec591c46bed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d79f1af503743878943a0d3ac3fe630 (127.0.98.1:37973), d97a29850cc548d18b882742057e11dd (127.0.98.3:45271)
I20260620 04:25:09.829548 2633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:09.825336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43703 (local address 127.0.98.1:37973)
0620 04:25:09.825459 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:09.825464 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:09.825555 (+ 91us) server_negotiation.cc:407] Connection header received
0620 04:25:09.825674 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:09.825676 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:09.825729 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:09.825818 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:25:09.826079 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:09.826916 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:09.828501 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:09.829082 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:09.829184 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:09.829247 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:09.829366 (+ 119us) server_negotiation.cc:299] Negotiation successful
0620 04:25:09.829440 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 04:25:09.829552 2366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:09.825263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37973 (local address 127.0.98.2:43703)
0620 04:25:09.825457 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:09.825471 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:09.825565 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:09.825833 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:09.825835 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:09.825847 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:25:09.825955 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:09.825962 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:09.827025 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:09.827028 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:09.828374 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:09.828385 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:09.829183 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:09.829186 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:09.829259 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:09.829278 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 04:25:09.829331 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 04:25:09.829397 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"mutex_wait_us":18}
I20260620 04:25:09.829787 2634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:09.825417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40335 (local address 127.0.98.3:45271)
0620 04:25:09.825551 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:09.825555 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:09.825733 (+ 178us) server_negotiation.cc:407] Connection header received
0620 04:25:09.825791 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:09.825794 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:09.825842 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:09.825911 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:25:09.826163 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:09.826937 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:09.828735 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:09.829311 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:09.829403 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:09.829454 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:09.829609 (+ 155us) server_negotiation.cc:299] Negotiation successful
0620 04:25:09.829675 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 04:25:09.829849 2645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:09.825312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45271 (local address 127.0.98.2:40335)
0620 04:25:09.825643 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:09.825655 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:09.825732 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:09.825935 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:09.825937 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:09.825950 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:25:09.826044 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:09.826049 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:09.827025 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:09.827028 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:09.828634 (+ 1606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:09.828641 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:09.829412 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:09.829415 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:09.829500 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:09.829519 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 04:25:09.829589 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 04:25:09.829724 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"mutex_wait_us":21,"thread_start_us":102,"threads_started":1}
I20260620 04:25:09.830147 2287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8d3334666464f27855f4ef3aa9f43a5" candidate_uuid: "d4cfed1c9d714d02a3797ec591c46bed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d79f1af503743878943a0d3ac3fe630" is_pre_election: true
I20260620 04:25:09.830440 2287 raft_consensus.cc:2454] T c8d3334666464f27855f4ef3aa9f43a5 P 3d79f1af503743878943a0d3ac3fe630 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4cfed1c9d714d02a3797ec591c46bed in term 0.
I20260620 04:25:09.830523 2567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8d3334666464f27855f4ef3aa9f43a5" candidate_uuid: "d4cfed1c9d714d02a3797ec591c46bed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d97a29850cc548d18b882742057e11dd" is_pre_election: true
I20260620 04:25:09.830785 2567 raft_consensus.cc:2454] T c8d3334666464f27855f4ef3aa9f43a5 P d97a29850cc548d18b882742057e11dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4cfed1c9d714d02a3797ec591c46bed in term 0.
I20260620 04:25:09.830904 2359 leader_election.cc:302] T c8d3334666464f27855f4ef3aa9f43a5 P d4cfed1c9d714d02a3797ec591c46bed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d79f1af503743878943a0d3ac3fe630, d4cfed1c9d714d02a3797ec591c46bed; no voters:
I20260620 04:25:09.831168 2641 raft_consensus.cc:2781] T c8d3334666464f27855f4ef3aa9f43a5 P d4cfed1c9d714d02a3797ec591c46bed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 04:25:09.831244 2641 raft_consensus.cc:482] T c8d3334666464f27855f4ef3aa9f43a5 P d4cfed1c9d714d02a3797ec591c46bed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 04:25:09.831281 2641 raft_consensus.cc:3037] T c8d3334666464f27855f4ef3aa9f43a5 P d4cfed1c9d714d02a3797ec591c46bed [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:25:09.832340 2641 raft_consensus.cc:504] T c8d3334666464f27855f4ef3aa9f43a5 P d4cfed1c9d714d02a3797ec591c46bed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4cfed1c9d714d02a3797ec591c46bed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38267 } } peers { permanent_uuid: "3d79f1af503743878943a0d3ac3fe630" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "d97a29850cc548d18b882742057e11dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45271 } }
I20260620 04:25:09.832638 2641 leader_election.cc:288] T c8d3334666464f27855f4ef3aa9f43a5 P d4cfed1c9d714d02a3797ec591c46bed [CANDIDATE]: Term 1 election: Requested vote from peers 3d79f1af503743878943a0d3ac3fe630 (127.0.98.1:37973), d97a29850cc548d18b882742057e11dd (127.0.98.3:45271)
I20260620 04:25:09.832919 2287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8d3334666464f27855f4ef3aa9f43a5" candidate_uuid: "d4cfed1c9d714d02a3797ec591c46bed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d79f1af503743878943a0d3ac3fe630"
I20260620 04:25:09.832913 2567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8d3334666464f27855f4ef3aa9f43a5" candidate_uuid: "d4cfed1c9d714d02a3797ec591c46bed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d97a29850cc548d18b882742057e11dd"
I20260620 04:25:09.833022 2567 raft_consensus.cc:3037] T c8d3334666464f27855f4ef3aa9f43a5 P d97a29850cc548d18b882742057e11dd [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:25:09.833022 2287 raft_consensus.cc:3037] T c8d3334666464f27855f4ef3aa9f43a5 P 3d79f1af503743878943a0d3ac3fe630 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:25:09.834174 2287 raft_consensus.cc:2454] T c8d3334666464f27855f4ef3aa9f43a5 P 3d79f1af503743878943a0d3ac3fe630 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4cfed1c9d714d02a3797ec591c46bed in term 1.
I20260620 04:25:09.834432 2567 raft_consensus.cc:2454] T c8d3334666464f27855f4ef3aa9f43a5 P d97a29850cc548d18b882742057e11dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4cfed1c9d714d02a3797ec591c46bed in term 1.
I20260620 04:25:09.834442 2359 leader_election.cc:302] T c8d3334666464f27855f4ef3aa9f43a5 P d4cfed1c9d714d02a3797ec591c46bed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d79f1af503743878943a0d3ac3fe630, d4cfed1c9d714d02a3797ec591c46bed; no voters:
I20260620 04:25:09.834589 2641 raft_consensus.cc:2781] T c8d3334666464f27855f4ef3aa9f43a5 P d4cfed1c9d714d02a3797ec591c46bed [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:25:09.834796 2641 raft_consensus.cc:686] T c8d3334666464f27855f4ef3aa9f43a5 P d4cfed1c9d714d02a3797ec591c46bed [term 1 LEADER]: Becoming Leader. State: Replica: d4cfed1c9d714d02a3797ec591c46bed, State: Running, Role: LEADER
I20260620 04:25:09.835125 2641 consensus_queue.cc:237] T c8d3334666464f27855f4ef3aa9f43a5 P d4cfed1c9d714d02a3797ec591c46bed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4cfed1c9d714d02a3797ec591c46bed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38267 } } peers { permanent_uuid: "3d79f1af503743878943a0d3ac3fe630" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "d97a29850cc548d18b882742057e11dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45271 } }
I20260620 04:25:09.838259 2128 catalog_manager.cc:5495] T c8d3334666464f27855f4ef3aa9f43a5 P d4cfed1c9d714d02a3797ec591c46bed reported cstate change: term changed from 0 to 1, leader changed from <none> to d4cfed1c9d714d02a3797ec591c46bed (127.0.98.2). New cstate: current_term: 1 leader_uuid: "d4cfed1c9d714d02a3797ec591c46bed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4cfed1c9d714d02a3797ec591c46bed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d79f1af503743878943a0d3ac3fe630" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d97a29850cc548d18b882742057e11dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45271 } health_report { overall_health: UNKNOWN } } }
W20260620 04:25:09.862725 2614 tablet.cc:2369] T c8d3334666464f27855f4ef3aa9f43a5 P d97a29850cc548d18b882742057e11dd: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 04:25:09 dist-test-slave-nk6d krb5kdc[2085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929508, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:25:09.873298 2179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:09.865628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38078 (local address 127.0.98.62:34081)
0620 04:25:09.865758 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:09.865763 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:09.865943 (+ 180us) server_negotiation.cc:407] Connection header received
0620 04:25:09.866042 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:09.866044 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:09.866083 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:09.866147 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:09.866768 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:09.867298 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:09.868109 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:09.868322 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:09.868418 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:09.870468 (+ 2050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:09.870471 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:09.870473 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:25:09.870492 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:09.871730 (+ 1238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:09.872214 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:09.872217 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:09.872218 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:09.872282 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:09.872501 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:09.872504 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:09.872506 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:09.872660 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:09.872740 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:09.873015 (+ 275us) server_negotiation.cc:299] Negotiation successful
0620 04:25:09.873166 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":57,"server-negotiator.queue_time_us":53}
I20260620 04:25:09.873298 2653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:09.865574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34081 (local address 127.0.0.1:38078)
0620 04:25:09.865848 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:09.865861 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:09.865972 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:09.866173 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:09.866175 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:09.866575 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:09.866667 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:09.866673 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:09.867384 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:09.867387 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:09.868020 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:09.868029 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:09.868423 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:09.868426 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:09.868491 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:09.868994 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:09.869005 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:09.870344 (+ 1339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:25:09.871822 (+ 1478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:09.871831 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:09.871838 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:09.872125 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:09.872353 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:09.872356 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:09.872358 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:09.872431 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:09.872751 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:25:09.872759 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:25:09.872864 (+ 105us) client_negotiation.cc:769] Sending connection context
0620 04:25:09.872963 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 04:25:09.873113 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":86,"threads_started":1}
I20260620 04:25:09.877048 2128 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38078:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 04:25:09.877725 2128 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 04:25:09.882264 2267 tablet_service.cc:1467] Processing CreateTablet for tablet 9be831445391407f8357307022020fd6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=cb90cc1e891c434eb4cedfdf014c98c5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 04:25:09.882262 2547 tablet_service.cc:1467] Processing CreateTablet for tablet 9be831445391407f8357307022020fd6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=cb90cc1e891c434eb4cedfdf014c98c5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 04:25:09.882251 2407 tablet_service.cc:1467] Processing CreateTablet for tablet 9be831445391407f8357307022020fd6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=cb90cc1e891c434eb4cedfdf014c98c5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 04:25:09.882900 2267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9be831445391407f8357307022020fd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:25:09.882905 2407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9be831445391407f8357307022020fd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:25:09.882915 2547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9be831445391407f8357307022020fd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:25:09.885430 2636 tablet_bootstrap.cc:492] T 9be831445391407f8357307022020fd6 P 3d79f1af503743878943a0d3ac3fe630: Bootstrap starting.
I20260620 04:25:09.885689 2637 tablet_bootstrap.cc:492] T 9be831445391407f8357307022020fd6 P d4cfed1c9d714d02a3797ec591c46bed: Bootstrap starting.
I20260620 04:25:09.886449 2636 tablet_bootstrap.cc:654] T 9be831445391407f8357307022020fd6 P 3d79f1af503743878943a0d3ac3fe630: Neither blocks nor log segments found. Creating new log.
I20260620 04:25:09.886674 2637 tablet_bootstrap.cc:654] T 9be831445391407f8357307022020fd6 P d4cfed1c9d714d02a3797ec591c46bed: Neither blocks nor log segments found. Creating new log.
I20260620 04:25:09.887241 2640 tablet_bootstrap.cc:492] T 9be831445391407f8357307022020fd6 P d97a29850cc548d18b882742057e11dd: Bootstrap starting.
I20260620 04:25:09.887607 2636 tablet_bootstrap.cc:492] T 9be831445391407f8357307022020fd6 P 3d79f1af503743878943a0d3ac3fe630: No bootstrap required, opened a new log
I20260620 04:25:09.887698 2636 ts_tablet_manager.cc:1397] T 9be831445391407f8357307022020fd6 P 3d79f1af503743878943a0d3ac3fe630: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 04:25:09.888177 2640 tablet_bootstrap.cc:654] T 9be831445391407f8357307022020fd6 P d97a29850cc548d18b882742057e11dd: Neither blocks nor log segments found. Creating new log.
I20260620 04:25:09.888253 2636 raft_consensus.cc:348] T 9be831445391407f8357307022020fd6 P 3d79f1af503743878943a0d3ac3fe630 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d79f1af503743878943a0d3ac3fe630" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "d4cfed1c9d714d02a3797ec591c46bed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38267 } } peers { permanent_uuid: "d97a29850cc548d18b882742057e11dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45271 } }
I20260620 04:25:09.888367 2636 raft_consensus.cc:374] T 9be831445391407f8357307022020fd6 P 3d79f1af503743878943a0d3ac3fe630 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:25:09.888406 2636 raft_consensus.cc:729] T 9be831445391407f8357307022020fd6 P 3d79f1af503743878943a0d3ac3fe630 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d79f1af503743878943a0d3ac3fe630, State: Initialized, Role: FOLLOWER
I20260620 04:25:09.888531 2636 consensus_queue.cc:260] T 9be831445391407f8357307022020fd6 P 3d79f1af503743878943a0d3ac3fe630 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d79f1af503743878943a0d3ac3fe630" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "d4cfed1c9d714d02a3797ec591c46bed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38267 } } peers { permanent_uuid: "d97a29850cc548d18b882742057e11dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45271 } }
I20260620 04:25:09.888785 2636 ts_tablet_manager.cc:1428] T 9be831445391407f8357307022020fd6 P 3d79f1af503743878943a0d3ac3fe630: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 04:25:09.888792 2642 tablet_replica.cc:439] T 9be831445391407f8357307022020fd6 P 3d79f1af503743878943a0d3ac3fe630: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d79f1af503743878943a0d3ac3fe630" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "d4cfed1c9d714d02a3797ec591c46bed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38267 } } peers { permanent_uuid: "d97a29850cc548d18b882742057e11dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45271 } } }
I20260620 04:25:09.888942 2642 tablet_replica.cc:442] T 9be831445391407f8357307022020fd6 P 3d79f1af503743878943a0d3ac3fe630: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 04:25:09.889113 2637 tablet_bootstrap.cc:492] T 9be831445391407f8357307022020fd6 P d4cfed1c9d714d02a3797ec591c46bed: No bootstrap required, opened a new log
I20260620 04:25:09.889173 2637 ts_tablet_manager.cc:1397] T 9be831445391407f8357307022020fd6 P d4cfed1c9d714d02a3797ec591c46bed: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260620 04:25:09.889535 2637 raft_consensus.cc:348] T 9be831445391407f8357307022020fd6 P d4cfed1c9d714d02a3797ec591c46bed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d79f1af503743878943a0d3ac3fe630" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "d4cfed1c9d714d02a3797ec591c46bed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38267 } } peers { permanent_uuid: "d97a29850cc548d18b882742057e11dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45271 } }
I20260620 04:25:09.889624 2637 raft_consensus.cc:374] T 9be831445391407f8357307022020fd6 P d4cfed1c9d714d02a3797ec591c46bed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:25:09.889652 2637 raft_consensus.cc:729] T 9be831445391407f8357307022020fd6 P d4cfed1c9d714d02a3797ec591c46bed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4cfed1c9d714d02a3797ec591c46bed, State: Initialized, Role: FOLLOWER
I20260620 04:25:09.889766 2637 consensus_queue.cc:260] T 9be831445391407f8357307022020fd6 P d4cfed1c9d714d02a3797ec591c46bed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d79f1af503743878943a0d3ac3fe630" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "d4cfed1c9d714d02a3797ec591c46bed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38267 } } peers { permanent_uuid: "d97a29850cc548d18b882742057e11dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45271 } }
I20260620 04:25:09.890013 2637 ts_tablet_manager.cc:1428] T 9be831445391407f8357307022020fd6 P d4cfed1c9d714d02a3797ec591c46bed: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 04:25:09.890235 2641 tablet_replica.cc:439] T 9be831445391407f8357307022020fd6 P d4cfed1c9d714d02a3797ec591c46bed: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d79f1af503743878943a0d3ac3fe630" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "d4cfed1c9d714d02a3797ec591c46bed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38267 } } peers { permanent_uuid: "d97a29850cc548d18b882742057e11dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45271 } } }
I20260620 04:25:09.890373 2641 tablet_replica.cc:442] T 9be831445391407f8357307022020fd6 P d4cfed1c9d714d02a3797ec591c46bed: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 04:25:09.890928 2640 tablet_bootstrap.cc:492] T 9be831445391407f8357307022020fd6 P d97a29850cc548d18b882742057e11dd: No bootstrap required, opened a new log
I20260620 04:25:09.891002 2640 ts_tablet_manager.cc:1397] T 9be831445391407f8357307022020fd6 P d97a29850cc548d18b882742057e11dd: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260620 04:25:09.891360 2640 raft_consensus.cc:348] T 9be831445391407f8357307022020fd6 P d97a29850cc548d18b882742057e11dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d79f1af503743878943a0d3ac3fe630" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "d4cfed1c9d714d02a3797ec591c46bed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38267 } } peers { permanent_uuid: "d97a29850cc548d18b882742057e11dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45271 } }
I20260620 04:25:09.891451 2640 raft_consensus.cc:374] T 9be831445391407f8357307022020fd6 P d97a29850cc548d18b882742057e11dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:25:09.891480 2640 raft_consensus.cc:729] T 9be831445391407f8357307022020fd6 P d97a29850cc548d18b882742057e11dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d97a29850cc548d18b882742057e11dd, State: Initialized, Role: FOLLOWER
I20260620 04:25:09.891594 2640 consensus_queue.cc:260] T 9be831445391407f8357307022020fd6 P d97a29850cc548d18b882742057e11dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d79f1af503743878943a0d3ac3fe630" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "d4cfed1c9d714d02a3797ec591c46bed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38267 } } peers { permanent_uuid: "d97a29850cc548d18b882742057e11dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45271 } }
I20260620 04:25:09.891858 2640 ts_tablet_manager.cc:1428] T 9be831445391407f8357307022020fd6 P d97a29850cc548d18b882742057e11dd: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 04:25:09.891834 2644 tablet_replica.cc:439] T 9be831445391407f8357307022020fd6 P d97a29850cc548d18b882742057e11dd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d79f1af503743878943a0d3ac3fe630" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "d4cfed1c9d714d02a3797ec591c46bed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38267 } } peers { permanent_uuid: "d97a29850cc548d18b882742057e11dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45271 } } }
I20260620 04:25:09.892000 2644 tablet_replica.cc:442] T 9be831445391407f8357307022020fd6 P d97a29850cc548d18b882742057e11dd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 04:25:09.893867 2642 raft_consensus.cc:482] T 9be831445391407f8357307022020fd6 P 3d79f1af503743878943a0d3ac3fe630 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:25:09.893975 2642 raft_consensus.cc:504] T 9be831445391407f8357307022020fd6 P 3d79f1af503743878943a0d3ac3fe630 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d79f1af503743878943a0d3ac3fe630" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "d4cfed1c9d714d02a3797ec591c46bed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38267 } } peers { permanent_uuid: "d97a29850cc548d18b882742057e11dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45271 } }
I20260620 04:25:09.894994 2642 leader_election.cc:288] T 9be831445391407f8357307022020fd6 P 3d79f1af503743878943a0d3ac3fe630 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4cfed1c9d714d02a3797ec591c46bed (127.0.98.2:38267), d97a29850cc548d18b882742057e11dd (127.0.98.3:45271)
I20260620 04:25:09.898550 2657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:09.895177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45271 (local address 127.0.98.1:43847)
0620 04:25:09.895394 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:09.895417 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:09.895506 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:09.895751 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:09.895754 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:09.895770 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:25:09.895894 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:09.895901 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:09.896749 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:09.896753 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:09.897550 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:09.897559 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:09.898156 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:09.898160 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:09.898264 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:09.898282 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 04:25:09.898343 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 04:25:09.898416 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":76,"threads_started":1}
I20260620 04:25:09.898541 2634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:09.895468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43847 (local address 127.0.98.3:45271)
0620 04:25:09.895576 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:09.895580 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:09.895592 (+ 12us) server_negotiation.cc:407] Connection header received
0620 04:25:09.895629 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:09.895633 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:09.895670 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:09.895736 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:25:09.896017 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:09.896657 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:09.897657 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:09.898041 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:09.898126 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:09.898184 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:09.898364 (+ 180us) server_negotiation.cc:299] Negotiation successful
0620 04:25:09.898425 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 04:25:09.898876 2632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:09.895331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56805 (local address 127.0.98.2:38267)
0620 04:25:09.895442 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:09.895446 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:09.895604 (+ 158us) server_negotiation.cc:407] Connection header received
0620 04:25:09.895697 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:09.895699 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:09.895742 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:09.895807 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:25:09.896511 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:09.897042 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:09.897975 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:09.898360 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:09.898446 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:09.898498 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:09.898688 (+ 190us) server_negotiation.cc:299] Negotiation successful
0620 04:25:09.898756 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 04:25:09.898993 2567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9be831445391407f8357307022020fd6" candidate_uuid: "3d79f1af503743878943a0d3ac3fe630" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d97a29850cc548d18b882742057e11dd" is_pre_election: true
I20260620 04:25:09.899114 2567 raft_consensus.cc:2454] T 9be831445391407f8357307022020fd6 P d97a29850cc548d18b882742057e11dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d79f1af503743878943a0d3ac3fe630 in term 0.
I20260620 04:25:09.899199 2658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:09.895177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38267 (local address 127.0.98.1:56805)
0620 04:25:09.895518 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:09.895530 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:09.895627 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:09.895872 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:09.895874 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:09.895888 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:25:09.896375 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:09.896385 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:09.897140 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:09.897144 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:09.897870 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:09.897878 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:09.898458 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:09.898462 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:09.898555 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:09.898575 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 04:25:09.898995 (+ 420us) client_negotiation.cc:240] Negotiation successful
0620 04:25:09.899058 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"mutex_wait_us":34,"thread_start_us":69,"threads_started":1}
I20260620 04:25:09.899446 2221 leader_election.cc:302] T 9be831445391407f8357307022020fd6 P 3d79f1af503743878943a0d3ac3fe630 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d79f1af503743878943a0d3ac3fe630, d97a29850cc548d18b882742057e11dd; no voters:
I20260620 04:25:09.899667 2642 raft_consensus.cc:2781] T 9be831445391407f8357307022020fd6 P 3d79f1af503743878943a0d3ac3fe630 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 04:25:09.899657 2427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9be831445391407f8357307022020fd6" candidate_uuid: "3d79f1af503743878943a0d3ac3fe630" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4cfed1c9d714d02a3797ec591c46bed" is_pre_election: true
I20260620 04:25:09.899724 2642 raft_consensus.cc:482] T 9be831445391407f8357307022020fd6 P 3d79f1af503743878943a0d3ac3fe630 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 04:25:09.899757 2642 raft_consensus.cc:3037] T 9be831445391407f8357307022020fd6 P 3d79f1af503743878943a0d3ac3fe630 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:25:09.899863 2427 raft_consensus.cc:2454] T 9be831445391407f8357307022020fd6 P d4cfed1c9d714d02a3797ec591c46bed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d79f1af503743878943a0d3ac3fe630 in term 0.
I20260620 04:25:09.900813 2642 raft_consensus.cc:504] T 9be831445391407f8357307022020fd6 P 3d79f1af503743878943a0d3ac3fe630 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d79f1af503743878943a0d3ac3fe630" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "d4cfed1c9d714d02a3797ec591c46bed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38267 } } peers { permanent_uuid: "d97a29850cc548d18b882742057e11dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45271 } }
I20260620 04:25:09.901135 2642 leader_election.cc:288] T 9be831445391407f8357307022020fd6 P 3d79f1af503743878943a0d3ac3fe630 [CANDIDATE]: Term 1 election: Requested vote from peers d4cfed1c9d714d02a3797ec591c46bed (127.0.98.2:38267), d97a29850cc548d18b882742057e11dd (127.0.98.3:45271)
I20260620 04:25:09.901396 2427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9be831445391407f8357307022020fd6" candidate_uuid: "3d79f1af503743878943a0d3ac3fe630" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4cfed1c9d714d02a3797ec591c46bed"
I20260620 04:25:09.901429 2567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9be831445391407f8357307022020fd6" candidate_uuid: "3d79f1af503743878943a0d3ac3fe630" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d97a29850cc548d18b882742057e11dd"
I20260620 04:25:09.901486 2427 raft_consensus.cc:3037] T 9be831445391407f8357307022020fd6 P d4cfed1c9d714d02a3797ec591c46bed [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:25:09.901530 2567 raft_consensus.cc:3037] T 9be831445391407f8357307022020fd6 P d97a29850cc548d18b882742057e11dd [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:25:09.902323 2427 raft_consensus.cc:2454] T 9be831445391407f8357307022020fd6 P d4cfed1c9d714d02a3797ec591c46bed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d79f1af503743878943a0d3ac3fe630 in term 1.
I20260620 04:25:09.902387 2567 raft_consensus.cc:2454] T 9be831445391407f8357307022020fd6 P d97a29850cc548d18b882742057e11dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d79f1af503743878943a0d3ac3fe630 in term 1.
I20260620 04:25:09.902570 2220 leader_election.cc:302] T 9be831445391407f8357307022020fd6 P 3d79f1af503743878943a0d3ac3fe630 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d79f1af503743878943a0d3ac3fe630, d4cfed1c9d714d02a3797ec591c46bed; no voters:
I20260620 04:25:09.902735 2642 raft_consensus.cc:2781] T 9be831445391407f8357307022020fd6 P 3d79f1af503743878943a0d3ac3fe630 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:25:09.902915 2642 raft_consensus.cc:686] T 9be831445391407f8357307022020fd6 P 3d79f1af503743878943a0d3ac3fe630 [term 1 LEADER]: Becoming Leader. State: Replica: 3d79f1af503743878943a0d3ac3fe630, State: Running, Role: LEADER
I20260620 04:25:09.903218 2642 consensus_queue.cc:237] T 9be831445391407f8357307022020fd6 P 3d79f1af503743878943a0d3ac3fe630 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d79f1af503743878943a0d3ac3fe630" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "d4cfed1c9d714d02a3797ec591c46bed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38267 } } peers { permanent_uuid: "d97a29850cc548d18b882742057e11dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45271 } }
W20260620 04:25:09.905057 2474 tablet.cc:2369] T 9be831445391407f8357307022020fd6 P d4cfed1c9d714d02a3797ec591c46bed: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 04:25:09.905721 2659 tablet_replica.cc:439] T 9be831445391407f8357307022020fd6 P 3d79f1af503743878943a0d3ac3fe630: TxnStatusTablet state changed. Reason: New leader 3d79f1af503743878943a0d3ac3fe630. Latest consensus state: current_term: 1 leader_uuid: "3d79f1af503743878943a0d3ac3fe630" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d79f1af503743878943a0d3ac3fe630" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "d4cfed1c9d714d02a3797ec591c46bed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38267 } } peers { permanent_uuid: "d97a29850cc548d18b882742057e11dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45271 } } }
I20260620 04:25:09.905833 2659 tablet_replica.cc:442] T 9be831445391407f8357307022020fd6 P 3d79f1af503743878943a0d3ac3fe630: This TxnStatusTablet replica's current role is: LEADER
I20260620 04:25:09.906073 2661 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 04:25:09.906033 2129 catalog_manager.cc:5495] T 9be831445391407f8357307022020fd6 P 3d79f1af503743878943a0d3ac3fe630 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d79f1af503743878943a0d3ac3fe630 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "3d79f1af503743878943a0d3ac3fe630" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d79f1af503743878943a0d3ac3fe630" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4cfed1c9d714d02a3797ec591c46bed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d97a29850cc548d18b882742057e11dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45271 } health_report { overall_health: UNKNOWN } } }
I20260620 04:25:09.906217 2661 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 04:25:09.922317 2633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:09.918129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41506 (local address 127.0.98.1:37973)
0620 04:25:09.918264 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:09.918268 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:09.918283 (+ 15us) server_negotiation.cc:407] Connection header received
0620 04:25:09.918350 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:09.918353 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:09.918421 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:09.918519 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:25:09.918757 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:09.919536 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:09.920760 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:09.921101 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:09.921203 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:09.921473 (+ 270us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:25:09.921948 (+ 475us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:25:09.922052 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:09.922117 (+ 65us) server_negotiation.cc:299] Negotiation successful
0620 04:25:09.922172 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 04:25:09.922312 2653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:09.917698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37973 (local address 127.0.0.1:41506)
0620 04:25:09.917849 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:09.917864 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:09.917957 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:09.918546 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:09.918549 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:09.918558 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 04:25:09.918661 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:09.918666 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:09.919638 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:09.919642 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:09.920644 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:09.920654 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:09.921207 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:09.921210 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:09.921302 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:09.921350 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 04:25:09.922039 (+ 689us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 04:25:09.922050 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 04:25:09.922107 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 04:25:09.922186 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 04:25:09.927573 2427 raft_consensus.cc:1261] T 9be831445391407f8357307022020fd6 P d4cfed1c9d714d02a3797ec591c46bed [term 1 FOLLOWER]: Refusing update from remote peer 3d79f1af503743878943a0d3ac3fe630: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:25:09.927619 2567 raft_consensus.cc:1261] T 9be831445391407f8357307022020fd6 P d97a29850cc548d18b882742057e11dd [term 1 FOLLOWER]: Refusing update from remote peer 3d79f1af503743878943a0d3ac3fe630: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:25:09.928153 2659 consensus_queue.cc:1035] T 9be831445391407f8357307022020fd6 P 3d79f1af503743878943a0d3ac3fe630 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4cfed1c9d714d02a3797ec591c46bed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:25:09.928351 2642 consensus_queue.cc:1035] T 9be831445391407f8357307022020fd6 P 3d79f1af503743878943a0d3ac3fe630 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d97a29850cc548d18b882742057e11dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:25:09.931375 2641 tablet_replica.cc:439] T 9be831445391407f8357307022020fd6 P d4cfed1c9d714d02a3797ec591c46bed: TxnStatusTablet state changed. Reason: New leader 3d79f1af503743878943a0d3ac3fe630. Latest consensus state: current_term: 1 leader_uuid: "3d79f1af503743878943a0d3ac3fe630" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d79f1af503743878943a0d3ac3fe630" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "d4cfed1c9d714d02a3797ec591c46bed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38267 } } peers { permanent_uuid: "d97a29850cc548d18b882742057e11dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45271 } } }
I20260620 04:25:09.931494 2641 tablet_replica.cc:442] T 9be831445391407f8357307022020fd6 P d4cfed1c9d714d02a3797ec591c46bed: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 04:25:09.931905 2644 tablet_replica.cc:439] T 9be831445391407f8357307022020fd6 P d97a29850cc548d18b882742057e11dd: TxnStatusTablet state changed. Reason: New leader 3d79f1af503743878943a0d3ac3fe630. Latest consensus state: current_term: 1 leader_uuid: "3d79f1af503743878943a0d3ac3fe630" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d79f1af503743878943a0d3ac3fe630" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "d4cfed1c9d714d02a3797ec591c46bed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38267 } } peers { permanent_uuid: "d97a29850cc548d18b882742057e11dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45271 } } }
I20260620 04:25:09.932049 2644 tablet_replica.cc:442] T 9be831445391407f8357307022020fd6 P d97a29850cc548d18b882742057e11dd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 04:25:09.935164 2659 tablet_replica.cc:439] T 9be831445391407f8357307022020fd6 P 3d79f1af503743878943a0d3ac3fe630: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3d79f1af503743878943a0d3ac3fe630" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d79f1af503743878943a0d3ac3fe630" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "d4cfed1c9d714d02a3797ec591c46bed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38267 } } peers { permanent_uuid: "d97a29850cc548d18b882742057e11dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45271 } } }
I20260620 04:25:09.935276 2659 tablet_replica.cc:442] T 9be831445391407f8357307022020fd6 P 3d79f1af503743878943a0d3ac3fe630: This TxnStatusTablet replica's current role is: LEADER
I20260620 04:25:09.935369 2664 mvcc.cc:205] Tried to move back new op lower bound from 7298783272656793600 to 7298783272565768192. Current Snapshot: MvccSnapshot[applied={T|T < 7298783272656793600}]
I20260620 04:25:09.935933 2641 tablet_replica.cc:439] T 9be831445391407f8357307022020fd6 P d4cfed1c9d714d02a3797ec591c46bed: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3d79f1af503743878943a0d3ac3fe630" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d79f1af503743878943a0d3ac3fe630" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "d4cfed1c9d714d02a3797ec591c46bed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38267 } } peers { permanent_uuid: "d97a29850cc548d18b882742057e11dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45271 } } }
I20260620 04:25:09.936057 2641 tablet_replica.cc:442] T 9be831445391407f8357307022020fd6 P d4cfed1c9d714d02a3797ec591c46bed: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 04:25:09.936565 2659 tablet_replica.cc:439] T 9be831445391407f8357307022020fd6 P 3d79f1af503743878943a0d3ac3fe630: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3d79f1af503743878943a0d3ac3fe630" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d79f1af503743878943a0d3ac3fe630" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "d4cfed1c9d714d02a3797ec591c46bed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38267 } } peers { permanent_uuid: "d97a29850cc548d18b882742057e11dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45271 } } }
I20260620 04:25:09.936662 2659 tablet_replica.cc:442] T 9be831445391407f8357307022020fd6 P 3d79f1af503743878943a0d3ac3fe630: This TxnStatusTablet replica's current role is: LEADER
I20260620 04:25:09.937769 2644 tablet_replica.cc:439] T 9be831445391407f8357307022020fd6 P d97a29850cc548d18b882742057e11dd: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3d79f1af503743878943a0d3ac3fe630" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d79f1af503743878943a0d3ac3fe630" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "d4cfed1c9d714d02a3797ec591c46bed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38267 } } peers { permanent_uuid: "d97a29850cc548d18b882742057e11dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45271 } } }
I20260620 04:25:09.937862 2644 tablet_replica.cc:442] T 9be831445391407f8357307022020fd6 P d97a29850cc548d18b882742057e11dd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 04:25:09.937935 2665 mvcc.cc:205] Tried to move back new op lower bound from 7298783272656793600 to 7298783272565768192. Current Snapshot: MvccSnapshot[applied={T|T < 7298783272656793600}]
I20260620 04:25:09.938464 2666 mvcc.cc:205] Tried to move back new op lower bound from 7298783272656793600 to 7298783272565768192. Current Snapshot: MvccSnapshot[applied={T|T < 7298783272656793600}]
W20260620 04:25:09.939615 2334 tablet.cc:2369] T c8d3334666464f27855f4ef3aa9f43a5 P 3d79f1af503743878943a0d3ac3fe630: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 04:25:09.948248 2632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:09.944610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47786 (local address 127.0.98.2:38267)
0620 04:25:09.944713 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:09.944716 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:09.944750 (+ 34us) server_negotiation.cc:407] Connection header received
0620 04:25:09.944895 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:09.944897 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:09.944940 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:09.945014 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:25:09.945243 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:09.945724 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:09.946889 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:09.947101 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:09.947189 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:09.947473 (+ 284us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:25:09.947893 (+ 420us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:25:09.948008 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:09.948072 (+ 64us) server_negotiation.cc:299] Negotiation successful
0620 04:25:09.948125 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 04:25:09.958252 2633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:09.954780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43211 (local address 127.0.98.1:37973)
0620 04:25:09.954910 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:09.954914 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:09.955114 (+ 200us) server_negotiation.cc:407] Connection header received
0620 04:25:09.955226 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:09.955229 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:09.955269 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:09.955326 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:25:09.955526 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:09.956090 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:09.957252 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:09.957449 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:09.957516 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:09.957889 (+ 373us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:25:09.957959 (+ 70us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:25:09.958024 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:09.958105 (+ 81us) server_negotiation.cc:299] Negotiation successful
0620 04:25:09.958149 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 04:25:09.958331 2675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:09.954703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37973 (local address 127.0.98.2:43211)
0620 04:25:09.955043 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:09.955057 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:09.955153 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:09.955345 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:09.955348 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:09.955354 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 04:25:09.955455 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:09.955461 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:09.956177 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:09.956181 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:09.957145 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:09.957164 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:09.957551 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:09.957554 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:09.957641 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:09.957688 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 04:25:09.958038 (+ 350us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 04:25:09.958043 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 04:25:09.958101 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 04:25:09.958176 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":79,"threads_started":1}
I20260620 04:25:09.964701 2567 raft_consensus.cc:1261] T c8d3334666464f27855f4ef3aa9f43a5 P d97a29850cc548d18b882742057e11dd [term 1 FOLLOWER]: Refusing update from remote peer d4cfed1c9d714d02a3797ec591c46bed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:25:09.965144 2287 raft_consensus.cc:1261] T c8d3334666464f27855f4ef3aa9f43a5 P 3d79f1af503743878943a0d3ac3fe630 [term 1 FOLLOWER]: Refusing update from remote peer d4cfed1c9d714d02a3797ec591c46bed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:25:09.965148 2641 consensus_queue.cc:1035] T c8d3334666464f27855f4ef3aa9f43a5 P d4cfed1c9d714d02a3797ec591c46bed [LEADER]: Connected to new peer: Peer: permanent_uuid: "d97a29850cc548d18b882742057e11dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:25:09.965590 2641 consensus_queue.cc:1035] T c8d3334666464f27855f4ef3aa9f43a5 P d4cfed1c9d714d02a3797ec591c46bed [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d79f1af503743878943a0d3ac3fe630" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:25:09.986824 2682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:09.982472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38267 (local address 127.0.98.1:51813)
0620 04:25:09.982859 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:09.982872 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:09.982977 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:09.983270 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:09.983273 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:09.983279 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 04:25:09.983408 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:09.983414 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:09.984366 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:09.984369 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:09.985604 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:09.985615 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:09.986073 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:09.986077 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:09.986167 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:09.986217 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 04:25:09.986509 (+ 292us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 04:25:09.986514 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 04:25:09.986566 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 04:25:09.986644 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":103,"threads_started":1}
I20260620 04:25:09.986827 2632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:09.982567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51813 (local address 127.0.98.2:38267)
0620 04:25:09.982678 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:09.982682 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:09.982958 (+ 276us) server_negotiation.cc:407] Connection header received
0620 04:25:09.983127 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:09.983130 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:09.983178 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:09.983254 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:25:09.983540 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:09.984263 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:09.985709 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:09.985963 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:09.986043 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:09.986358 (+ 315us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:25:09.986431 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:25:09.986498 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:09.986593 (+ 95us) server_negotiation.cc:299] Negotiation successful
0620 04:25:09.986665 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 04:25:10.013676 2129 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52038:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:25:10.013896 2129 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52038:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:25:10.016748 2129 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 15ec2623f54b4519af25addf17f1c427: Sending DeleteTablet for 3 replicas of tablet c8d3334666464f27855f4ef3aa9f43a5
I20260620 04:25:10.017463 2407 tablet_service.cc:1514] Processing DeleteTablet for tablet c8d3334666464f27855f4ef3aa9f43a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:25:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34240
I20260620 04:25:10.017993 2685 tablet_replica.cc:330] T c8d3334666464f27855f4ef3aa9f43a5 P d4cfed1c9d714d02a3797ec591c46bed: stopping tablet replica
I20260620 04:25:10.018077 2267 tablet_service.cc:1514] Processing DeleteTablet for tablet c8d3334666464f27855f4ef3aa9f43a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:25:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48964
I20260620 04:25:10.018311 2685 raft_consensus.cc:2229] T c8d3334666464f27855f4ef3aa9f43a5 P d4cfed1c9d714d02a3797ec591c46bed [term 1 LEADER]: Raft consensus shutting down.
I20260620 04:25:10.018683 2685 raft_consensus.cc:2258] T c8d3334666464f27855f4ef3aa9f43a5 P d4cfed1c9d714d02a3797ec591c46bed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:25:10.019136 2547 tablet_service.cc:1514] Processing DeleteTablet for tablet c8d3334666464f27855f4ef3aa9f43a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:25:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34728
I20260620 04:25:10.020390 2687 tablet_replica.cc:330] T c8d3334666464f27855f4ef3aa9f43a5 P d97a29850cc548d18b882742057e11dd: stopping tablet replica
I20260620 04:25:10.020382 2129 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52038:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 04:25:10.020829 2129 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 04:25:10.020969 2687 raft_consensus.cc:2229] T c8d3334666464f27855f4ef3aa9f43a5 P d97a29850cc548d18b882742057e11dd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:25:10.021248 2687 raft_consensus.cc:2258] T c8d3334666464f27855f4ef3aa9f43a5 P d97a29850cc548d18b882742057e11dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:25:10.021657 2686 tablet_replica.cc:330] T c8d3334666464f27855f4ef3aa9f43a5 P 3d79f1af503743878943a0d3ac3fe630: stopping tablet replica
I20260620 04:25:10.021967 2686 raft_consensus.cc:2229] T c8d3334666464f27855f4ef3aa9f43a5 P 3d79f1af503743878943a0d3ac3fe630 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:25:10.022181 2686 raft_consensus.cc:2258] T c8d3334666464f27855f4ef3aa9f43a5 P 3d79f1af503743878943a0d3ac3fe630 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:25:10.023870 2686 ts_tablet_manager.cc:1905] T c8d3334666464f27855f4ef3aa9f43a5 P 3d79f1af503743878943a0d3ac3fe630: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:25:10.026947 2267 tablet_service.cc:1467] Processing CreateTablet for tablet 4745b4addcc54013b27ff99db8533e33 (DEFAULT_TABLE table=test-table [id=9302a2d0a69d436c9ffd7c4cd3562b1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:25:10.027199 2267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4745b4addcc54013b27ff99db8533e33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:25:10.027786 2686 ts_tablet_manager.cc:1918] T c8d3334666464f27855f4ef3aa9f43a5 P 3d79f1af503743878943a0d3ac3fe630: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 04:25:10.027884 2686 log.cc:1196] T c8d3334666464f27855f4ef3aa9f43a5 P 3d79f1af503743878943a0d3ac3fe630: Deleting WAL directory at /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929496330398-392-0/minicluster-data/ts-0/wal/wals/c8d3334666464f27855f4ef3aa9f43a5
I20260620 04:25:10.028196 2686 ts_tablet_manager.cc:1939] T c8d3334666464f27855f4ef3aa9f43a5 P 3d79f1af503743878943a0d3ac3fe630: Deleting consensus metadata
I20260620 04:25:10.028295 2687 ts_tablet_manager.cc:1905] T c8d3334666464f27855f4ef3aa9f43a5 P d97a29850cc548d18b882742057e11dd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:25:10.029354 2636 tablet_bootstrap.cc:492] T 4745b4addcc54013b27ff99db8533e33 P 3d79f1af503743878943a0d3ac3fe630: Bootstrap starting.
I20260620 04:25:10.029419 2115 catalog_manager.cc:4841] TS 3d79f1af503743878943a0d3ac3fe630 (127.0.98.1:37973): tablet c8d3334666464f27855f4ef3aa9f43a5 (table test-table [id=303b13f341704793aecca47d268cd87e]) successfully deleted
I20260620 04:25:10.030284 2636 tablet_bootstrap.cc:654] T 4745b4addcc54013b27ff99db8533e33 P 3d79f1af503743878943a0d3ac3fe630: Neither blocks nor log segments found. Creating new log.
I20260620 04:25:10.030989 2687 ts_tablet_manager.cc:1918] T c8d3334666464f27855f4ef3aa9f43a5 P d97a29850cc548d18b882742057e11dd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 04:25:10.031086 2687 log.cc:1196] T c8d3334666464f27855f4ef3aa9f43a5 P d97a29850cc548d18b882742057e11dd: Deleting WAL directory at /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929496330398-392-0/minicluster-data/ts-2/wal/wals/c8d3334666464f27855f4ef3aa9f43a5
I20260620 04:25:10.031306 2547 tablet_service.cc:1467] Processing CreateTablet for tablet 4745b4addcc54013b27ff99db8533e33 (DEFAULT_TABLE table=test-table [id=9302a2d0a69d436c9ffd7c4cd3562b1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:25:10.031545 2636 tablet_bootstrap.cc:492] T 4745b4addcc54013b27ff99db8533e33 P 3d79f1af503743878943a0d3ac3fe630: No bootstrap required, opened a new log
I20260620 04:25:10.031569 2547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4745b4addcc54013b27ff99db8533e33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:25:10.031610 2636 ts_tablet_manager.cc:1397] T 4745b4addcc54013b27ff99db8533e33 P 3d79f1af503743878943a0d3ac3fe630: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 04:25:10.032056 2636 raft_consensus.cc:348] T 4745b4addcc54013b27ff99db8533e33 P 3d79f1af503743878943a0d3ac3fe630 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d79f1af503743878943a0d3ac3fe630" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "d4cfed1c9d714d02a3797ec591c46bed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38267 } } peers { permanent_uuid: "d97a29850cc548d18b882742057e11dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45271 } }
I20260620 04:25:10.032159 2636 raft_consensus.cc:374] T 4745b4addcc54013b27ff99db8533e33 P 3d79f1af503743878943a0d3ac3fe630 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:25:10.032187 2636 raft_consensus.cc:729] T 4745b4addcc54013b27ff99db8533e33 P 3d79f1af503743878943a0d3ac3fe630 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d79f1af503743878943a0d3ac3fe630, State: Initialized, Role: FOLLOWER
I20260620 04:25:10.032294 2636 consensus_queue.cc:260] T 4745b4addcc54013b27ff99db8533e33 P 3d79f1af503743878943a0d3ac3fe630 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d79f1af503743878943a0d3ac3fe630" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "d4cfed1c9d714d02a3797ec591c46bed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38267 } } peers { permanent_uuid: "d97a29850cc548d18b882742057e11dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45271 } }
I20260620 04:25:10.032539 2636 ts_tablet_manager.cc:1428] T 4745b4addcc54013b27ff99db8533e33 P 3d79f1af503743878943a0d3ac3fe630: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 04:25:10.033561 2640 tablet_bootstrap.cc:492] T 4745b4addcc54013b27ff99db8533e33 P d97a29850cc548d18b882742057e11dd: Bootstrap starting.
I20260620 04:25:10.026960 2407 tablet_service.cc:1467] Processing CreateTablet for tablet 4745b4addcc54013b27ff99db8533e33 (DEFAULT_TABLE table=test-table [id=9302a2d0a69d436c9ffd7c4cd3562b1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:25:10.034230 2407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4745b4addcc54013b27ff99db8533e33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:25:10.034415 2640 tablet_bootstrap.cc:654] T 4745b4addcc54013b27ff99db8533e33 P d97a29850cc548d18b882742057e11dd: Neither blocks nor log segments found. Creating new log.
I20260620 04:25:10.031371 2687 ts_tablet_manager.cc:1939] T c8d3334666464f27855f4ef3aa9f43a5 P d97a29850cc548d18b882742057e11dd: Deleting consensus metadata
I20260620 04:25:10.036934 2637 tablet_bootstrap.cc:492] T 4745b4addcc54013b27ff99db8533e33 P d4cfed1c9d714d02a3797ec591c46bed: Bootstrap starting.
I20260620 04:25:10.037859 2637 tablet_bootstrap.cc:654] T 4745b4addcc54013b27ff99db8533e33 P d4cfed1c9d714d02a3797ec591c46bed: Neither blocks nor log segments found. Creating new log.
I20260620 04:25:10.037950 2117 catalog_manager.cc:4841] TS d97a29850cc548d18b882742057e11dd (127.0.98.3:45271): tablet c8d3334666464f27855f4ef3aa9f43a5 (table test-table [id=303b13f341704793aecca47d268cd87e]) successfully deleted
I20260620 04:25:10.020202 2685 ts_tablet_manager.cc:1905] T c8d3334666464f27855f4ef3aa9f43a5 P d4cfed1c9d714d02a3797ec591c46bed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:25:10.039186 2637 tablet_bootstrap.cc:492] T 4745b4addcc54013b27ff99db8533e33 P d4cfed1c9d714d02a3797ec591c46bed: No bootstrap required, opened a new log
I20260620 04:25:10.039266 2637 ts_tablet_manager.cc:1397] T 4745b4addcc54013b27ff99db8533e33 P d4cfed1c9d714d02a3797ec591c46bed: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 04:25:10.039706 2637 raft_consensus.cc:348] T 4745b4addcc54013b27ff99db8533e33 P d4cfed1c9d714d02a3797ec591c46bed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d79f1af503743878943a0d3ac3fe630" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "d4cfed1c9d714d02a3797ec591c46bed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38267 } } peers { permanent_uuid: "d97a29850cc548d18b882742057e11dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45271 } }
I20260620 04:25:10.039829 2637 raft_consensus.cc:374] T 4745b4addcc54013b27ff99db8533e33 P d4cfed1c9d714d02a3797ec591c46bed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:25:10.039860 2637 raft_consensus.cc:729] T 4745b4addcc54013b27ff99db8533e33 P d4cfed1c9d714d02a3797ec591c46bed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4cfed1c9d714d02a3797ec591c46bed, State: Initialized, Role: FOLLOWER
I20260620 04:25:10.039996 2637 consensus_queue.cc:260] T 4745b4addcc54013b27ff99db8533e33 P d4cfed1c9d714d02a3797ec591c46bed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d79f1af503743878943a0d3ac3fe630" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "d4cfed1c9d714d02a3797ec591c46bed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38267 } } peers { permanent_uuid: "d97a29850cc548d18b882742057e11dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45271 } }
I20260620 04:25:10.040257 2637 ts_tablet_manager.cc:1428] T 4745b4addcc54013b27ff99db8533e33 P d4cfed1c9d714d02a3797ec591c46bed: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 04:25:10.043818 2685 ts_tablet_manager.cc:1918] T c8d3334666464f27855f4ef3aa9f43a5 P d4cfed1c9d714d02a3797ec591c46bed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 04:25:10.043931 2685 log.cc:1196] T c8d3334666464f27855f4ef3aa9f43a5 P d4cfed1c9d714d02a3797ec591c46bed: Deleting WAL directory at /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929496330398-392-0/minicluster-data/ts-1/wal/wals/c8d3334666464f27855f4ef3aa9f43a5
I20260620 04:25:10.044272 2685 ts_tablet_manager.cc:1939] T c8d3334666464f27855f4ef3aa9f43a5 P d4cfed1c9d714d02a3797ec591c46bed: Deleting consensus metadata
I20260620 04:25:10.045502 2116 catalog_manager.cc:4841] TS d4cfed1c9d714d02a3797ec591c46bed (127.0.98.2:38267): tablet c8d3334666464f27855f4ef3aa9f43a5 (table test-table [id=303b13f341704793aecca47d268cd87e]) successfully deleted
I20260620 04:25:10.048929 2640 tablet_bootstrap.cc:492] T 4745b4addcc54013b27ff99db8533e33 P d97a29850cc548d18b882742057e11dd: No bootstrap required, opened a new log
I20260620 04:25:10.049054 2640 ts_tablet_manager.cc:1397] T 4745b4addcc54013b27ff99db8533e33 P d97a29850cc548d18b882742057e11dd: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.000s
I20260620 04:25:10.049899 2640 raft_consensus.cc:348] T 4745b4addcc54013b27ff99db8533e33 P d97a29850cc548d18b882742057e11dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d79f1af503743878943a0d3ac3fe630" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "d4cfed1c9d714d02a3797ec591c46bed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38267 } } peers { permanent_uuid: "d97a29850cc548d18b882742057e11dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45271 } }
I20260620 04:25:10.050046 2640 raft_consensus.cc:374] T 4745b4addcc54013b27ff99db8533e33 P d97a29850cc548d18b882742057e11dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:25:10.050087 2640 raft_consensus.cc:729] T 4745b4addcc54013b27ff99db8533e33 P d97a29850cc548d18b882742057e11dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d97a29850cc548d18b882742057e11dd, State: Initialized, Role: FOLLOWER
I20260620 04:25:10.050218 2640 consensus_queue.cc:260] T 4745b4addcc54013b27ff99db8533e33 P d97a29850cc548d18b882742057e11dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d79f1af503743878943a0d3ac3fe630" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "d4cfed1c9d714d02a3797ec591c46bed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38267 } } peers { permanent_uuid: "d97a29850cc548d18b882742057e11dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45271 } }
I20260620 04:25:10.050508 2640 ts_tablet_manager.cc:1428] T 4745b4addcc54013b27ff99db8533e33 P d97a29850cc548d18b882742057e11dd: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260620 04:25:10.074083 2644 raft_consensus.cc:482] T 4745b4addcc54013b27ff99db8533e33 P d97a29850cc548d18b882742057e11dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:25:10.074249 2644 raft_consensus.cc:504] T 4745b4addcc54013b27ff99db8533e33 P d97a29850cc548d18b882742057e11dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d79f1af503743878943a0d3ac3fe630" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "d4cfed1c9d714d02a3797ec591c46bed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38267 } } peers { permanent_uuid: "d97a29850cc548d18b882742057e11dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45271 } }
I20260620 04:25:10.075585 2644 leader_election.cc:288] T 4745b4addcc54013b27ff99db8533e33 P d97a29850cc548d18b882742057e11dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d79f1af503743878943a0d3ac3fe630 (127.0.98.1:37973), d4cfed1c9d714d02a3797ec591c46bed (127.0.98.2:38267)
I20260620 04:25:10.080716 2506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:10.075798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37973 (local address 127.0.98.3:40449)
0620 04:25:10.076328 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:10.076342 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:10.076438 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:10.076834 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:10.076838 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:10.076856 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:25:10.076967 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:10.076973 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:10.078061 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:10.078064 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:10.079320 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:10.079330 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:10.080300 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:10.080304 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:10.080408 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:10.080430 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 04:25:10.080493 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 04:25:10.080562 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460}
I20260620 04:25:10.081184 2633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:10.076075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40449 (local address 127.0.98.1:37973)
0620 04:25:10.076215 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:10.076220 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:10.076522 (+ 302us) server_negotiation.cc:407] Connection header received
0620 04:25:10.076581 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:10.076584 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:10.076630 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:10.076692 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:25:10.077090 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:10.077911 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:10.079456 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:10.080077 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:10.080172 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:10.080241 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:10.080993 (+ 752us) server_negotiation.cc:299] Negotiation successful
0620 04:25:10.081069 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 04:25:10.081583 2287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4745b4addcc54013b27ff99db8533e33" candidate_uuid: "d97a29850cc548d18b882742057e11dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d79f1af503743878943a0d3ac3fe630" is_pre_election: true
I20260620 04:25:10.081755 2287 raft_consensus.cc:2454] T 4745b4addcc54013b27ff99db8533e33 P 3d79f1af503743878943a0d3ac3fe630 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d97a29850cc548d18b882742057e11dd in term 0.
I20260620 04:25:10.082008 2632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:10.076396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59903 (local address 127.0.98.2:38267)
0620 04:25:10.076519 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:10.076522 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:10.077778 (+ 1256us) server_negotiation.cc:407] Connection header received
0620 04:25:10.078001 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:10.078006 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:10.078064 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:10.078122 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:25:10.078500 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:10.079212 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:10.080684 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:10.081197 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:10.081262 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:10.081322 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:10.081817 (+ 495us) server_negotiation.cc:299] Negotiation successful
0620 04:25:10.081893 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 04:25:10.082008 2691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:10.075932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38267 (local address 127.0.98.3:59903)
0620 04:25:10.077688 (+ 1756us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:10.077702 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:10.077856 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:10.078258 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:10.078261 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:10.078278 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:25:10.078388 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:10.078394 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:10.079351 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:10.079354 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:10.080548 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:10.080558 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:10.081447 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:10.081450 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:10.081662 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:10.081685 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 04:25:10.081791 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 04:25:10.081873 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1691,"thread_start_us":83,"threads_started":1}
I20260620 04:25:10.082162 2499 leader_election.cc:302] T 4745b4addcc54013b27ff99db8533e33 P d97a29850cc548d18b882742057e11dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d79f1af503743878943a0d3ac3fe630, d97a29850cc548d18b882742057e11dd; no voters:
I20260620 04:25:10.082458 2427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4745b4addcc54013b27ff99db8533e33" candidate_uuid: "d97a29850cc548d18b882742057e11dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4cfed1c9d714d02a3797ec591c46bed" is_pre_election: true
I20260620 04:25:10.082597 2644 raft_consensus.cc:2781] T 4745b4addcc54013b27ff99db8533e33 P d97a29850cc548d18b882742057e11dd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 04:25:10.082603 2427 raft_consensus.cc:2454] T 4745b4addcc54013b27ff99db8533e33 P d4cfed1c9d714d02a3797ec591c46bed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d97a29850cc548d18b882742057e11dd in term 0.
I20260620 04:25:10.082656 2644 raft_consensus.cc:482] T 4745b4addcc54013b27ff99db8533e33 P d97a29850cc548d18b882742057e11dd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 04:25:10.082700 2644 raft_consensus.cc:3037] T 4745b4addcc54013b27ff99db8533e33 P d97a29850cc548d18b882742057e11dd [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:25:10.083920 2644 raft_consensus.cc:504] T 4745b4addcc54013b27ff99db8533e33 P d97a29850cc548d18b882742057e11dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d79f1af503743878943a0d3ac3fe630" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "d4cfed1c9d714d02a3797ec591c46bed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38267 } } peers { permanent_uuid: "d97a29850cc548d18b882742057e11dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45271 } }
I20260620 04:25:10.084333 2644 leader_election.cc:288] T 4745b4addcc54013b27ff99db8533e33 P d97a29850cc548d18b882742057e11dd [CANDIDATE]: Term 1 election: Requested vote from peers 3d79f1af503743878943a0d3ac3fe630 (127.0.98.1:37973), d4cfed1c9d714d02a3797ec591c46bed (127.0.98.2:38267)
I20260620 04:25:10.084708 2427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4745b4addcc54013b27ff99db8533e33" candidate_uuid: "d97a29850cc548d18b882742057e11dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4cfed1c9d714d02a3797ec591c46bed"
I20260620 04:25:10.084860 2427 raft_consensus.cc:3037] T 4745b4addcc54013b27ff99db8533e33 P d4cfed1c9d714d02a3797ec591c46bed [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:25:10.085762 2427 raft_consensus.cc:2454] T 4745b4addcc54013b27ff99db8533e33 P d4cfed1c9d714d02a3797ec591c46bed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d97a29850cc548d18b882742057e11dd in term 1.
I20260620 04:25:10.086012 2287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4745b4addcc54013b27ff99db8533e33" candidate_uuid: "d97a29850cc548d18b882742057e11dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d79f1af503743878943a0d3ac3fe630"
I20260620 04:25:10.086048 2500 leader_election.cc:302] T 4745b4addcc54013b27ff99db8533e33 P d97a29850cc548d18b882742057e11dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4cfed1c9d714d02a3797ec591c46bed, d97a29850cc548d18b882742057e11dd; no voters:
I20260620 04:25:10.086136 2287 raft_consensus.cc:3037] T 4745b4addcc54013b27ff99db8533e33 P 3d79f1af503743878943a0d3ac3fe630 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:25:10.087211 2287 raft_consensus.cc:2454] T 4745b4addcc54013b27ff99db8533e33 P 3d79f1af503743878943a0d3ac3fe630 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d97a29850cc548d18b882742057e11dd in term 1.
I20260620 04:25:10.087399 2644 raft_consensus.cc:2781] T 4745b4addcc54013b27ff99db8533e33 P d97a29850cc548d18b882742057e11dd [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:25:10.087574 2644 raft_consensus.cc:686] T 4745b4addcc54013b27ff99db8533e33 P d97a29850cc548d18b882742057e11dd [term 1 LEADER]: Becoming Leader. State: Replica: d97a29850cc548d18b882742057e11dd, State: Running, Role: LEADER
I20260620 04:25:10.087842 2644 consensus_queue.cc:237] T 4745b4addcc54013b27ff99db8533e33 P d97a29850cc548d18b882742057e11dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d79f1af503743878943a0d3ac3fe630" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "d4cfed1c9d714d02a3797ec591c46bed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38267 } } peers { permanent_uuid: "d97a29850cc548d18b882742057e11dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45271 } }
I20260620 04:25:10.090318 2129 catalog_manager.cc:5495] T 4745b4addcc54013b27ff99db8533e33 P d97a29850cc548d18b882742057e11dd reported cstate change: term changed from 0 to 1, leader changed from <none> to d97a29850cc548d18b882742057e11dd (127.0.98.3). New cstate: current_term: 1 leader_uuid: "d97a29850cc548d18b882742057e11dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d79f1af503743878943a0d3ac3fe630" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4cfed1c9d714d02a3797ec591c46bed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d97a29850cc548d18b882742057e11dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45271 } health_report { overall_health: HEALTHY } } }
I20260620 04:25:10.112895 2634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:10.109648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56472 (local address 127.0.98.3:45271)
0620 04:25:10.109744 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:10.109749 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:10.109764 (+ 15us) server_negotiation.cc:407] Connection header received
0620 04:25:10.109942 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:10.109945 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:10.109997 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:10.110081 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:25:10.110294 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:10.110789 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:10.111573 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:10.111797 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:10.111882 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:10.112151 (+ 269us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:25:10.112530 (+ 379us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:25:10.112630 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:10.112704 (+ 74us) server_negotiation.cc:299] Negotiation successful
0620 04:25:10.112754 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 04:25:10.123291 2619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:10.119880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37973 (local address 127.0.98.3:38167)
0620 04:25:10.120032 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:10.120048 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:10.120124 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:10.120382 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:10.120384 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:10.120393 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 04:25:10.120501 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:10.120507 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:10.121484 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:10.121487 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:10.122116 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:10.122124 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:10.122639 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:10.122642 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:10.122710 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:10.122751 (+ 41us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 04:25:10.123040 (+ 289us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 04:25:10.123046 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 04:25:10.123113 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 04:25:10.123180 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260620 04:25:10.123329 2633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:10.119944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38167 (local address 127.0.98.1:37973)
0620 04:25:10.120088 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:10.120092 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:10.120107 (+ 15us) server_negotiation.cc:407] Connection header received
0620 04:25:10.120199 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:10.120203 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:10.120254 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:10.120337 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:25:10.120609 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:10.121372 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:10.122217 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:10.122549 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:10.122648 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:10.122858 (+ 210us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:25:10.122939 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:25:10.123008 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:10.123136 (+ 128us) server_negotiation.cc:299] Negotiation successful
0620 04:25:10.123192 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 04:25:10.129812 2427 raft_consensus.cc:1261] T 4745b4addcc54013b27ff99db8533e33 P d4cfed1c9d714d02a3797ec591c46bed [term 1 FOLLOWER]: Refusing update from remote peer d97a29850cc548d18b882742057e11dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:25:10.129833 2287 raft_consensus.cc:1261] T 4745b4addcc54013b27ff99db8533e33 P 3d79f1af503743878943a0d3ac3fe630 [term 1 FOLLOWER]: Refusing update from remote peer d97a29850cc548d18b882742057e11dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:25:10.130271 2644 consensus_queue.cc:1035] T 4745b4addcc54013b27ff99db8533e33 P d97a29850cc548d18b882742057e11dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d79f1af503743878943a0d3ac3fe630" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:25:10.130628 2644 consensus_queue.cc:1035] T 4745b4addcc54013b27ff99db8533e33 P d97a29850cc548d18b882742057e11dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4cfed1c9d714d02a3797ec591c46bed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:25:10.150259 2634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:10.146693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47037 (local address 127.0.98.3:45271)
0620 04:25:10.146895 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:10.146900 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:10.146911 (+ 11us) server_negotiation.cc:407] Connection header received
0620 04:25:10.146949 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:10.146951 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:10.146999 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:10.147079 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:25:10.147289 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:10.148040 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:10.149176 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:10.149465 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:10.149566 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:10.149841 (+ 275us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:25:10.149943 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:25:10.149998 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:10.150110 (+ 112us) server_negotiation.cc:299] Negotiation successful
0620 04:25:10.150154 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260620 04:25:10.150324 2682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:10.146486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45271 (local address 127.0.98.1:47037)
0620 04:25:10.146624 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:10.146641 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:10.146726 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:10.147100 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:10.147103 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:10.147111 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 04:25:10.147202 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:10.147207 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:10.148150 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:10.148153 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:10.149060 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:10.149071 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:10.149587 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:10.149591 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:10.149679 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:10.149714 (+ 35us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 04:25:10.150027 (+ 313us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 04:25:10.150034 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 04:25:10.150102 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 04:25:10.150188 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
W20260620 04:25:10.151067 2666 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 04:25:11.710386 2129 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52038:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:25:11.710534 2129 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52038:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:25:11.712808 2129 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 15ec2623f54b4519af25addf17f1c427: Sending DeleteTablet for 3 replicas of tablet 4745b4addcc54013b27ff99db8533e33
I20260620 04:25:11.713289 2267 tablet_service.cc:1514] Processing DeleteTablet for tablet 4745b4addcc54013b27ff99db8533e33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:25:11 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48964
I20260620 04:25:11.713397 2407 tablet_service.cc:1514] Processing DeleteTablet for tablet 4745b4addcc54013b27ff99db8533e33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:25:11 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34240
I20260620 04:25:11.713589 2547 tablet_service.cc:1514] Processing DeleteTablet for tablet 4745b4addcc54013b27ff99db8533e33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:25:11 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34728
I20260620 04:25:11.713769 2723 tablet_replica.cc:330] T 4745b4addcc54013b27ff99db8533e33 P d4cfed1c9d714d02a3797ec591c46bed: stopping tablet replica
I20260620 04:25:11.713896 2723 raft_consensus.cc:2229] T 4745b4addcc54013b27ff99db8533e33 P d4cfed1c9d714d02a3797ec591c46bed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:25:11.713909 2722 tablet_replica.cc:330] T 4745b4addcc54013b27ff99db8533e33 P 3d79f1af503743878943a0d3ac3fe630: stopping tablet replica
I20260620 04:25:11.714007 2723 raft_consensus.cc:2258] T 4745b4addcc54013b27ff99db8533e33 P d4cfed1c9d714d02a3797ec591c46bed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:25:11.714010 2722 raft_consensus.cc:2229] T 4745b4addcc54013b27ff99db8533e33 P 3d79f1af503743878943a0d3ac3fe630 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:25:11.714104 2722 raft_consensus.cc:2258] T 4745b4addcc54013b27ff99db8533e33 P 3d79f1af503743878943a0d3ac3fe630 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:25:11.714020 2724 tablet_replica.cc:330] T 4745b4addcc54013b27ff99db8533e33 P d97a29850cc548d18b882742057e11dd: stopping tablet replica
I20260620 04:25:11.714465 2724 raft_consensus.cc:2229] T 4745b4addcc54013b27ff99db8533e33 P d97a29850cc548d18b882742057e11dd [term 1 LEADER]: Raft consensus shutting down.
I20260620 04:25:11.714712 2723 ts_tablet_manager.cc:1905] T 4745b4addcc54013b27ff99db8533e33 P d4cfed1c9d714d02a3797ec591c46bed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:25:11.714859 2724 raft_consensus.cc:2258] T 4745b4addcc54013b27ff99db8533e33 P d97a29850cc548d18b882742057e11dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:25:11.715752 2724 ts_tablet_manager.cc:1905] T 4745b4addcc54013b27ff99db8533e33 P d97a29850cc548d18b882742057e11dd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:25:11.717067 2722 ts_tablet_manager.cc:1905] T 4745b4addcc54013b27ff99db8533e33 P 3d79f1af503743878943a0d3ac3fe630: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:25:11.717530 2723 ts_tablet_manager.cc:1918] T 4745b4addcc54013b27ff99db8533e33 P d4cfed1c9d714d02a3797ec591c46bed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 04:25:11.717602 2723 log.cc:1196] T 4745b4addcc54013b27ff99db8533e33 P d4cfed1c9d714d02a3797ec591c46bed: Deleting WAL directory at /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929496330398-392-0/minicluster-data/ts-1/wal/wals/4745b4addcc54013b27ff99db8533e33
I20260620 04:25:11.717844 2723 ts_tablet_manager.cc:1939] T 4745b4addcc54013b27ff99db8533e33 P d4cfed1c9d714d02a3797ec591c46bed: Deleting consensus metadata
I20260620 04:25:11.718510 2116 catalog_manager.cc:4841] TS d4cfed1c9d714d02a3797ec591c46bed (127.0.98.2:38267): tablet 4745b4addcc54013b27ff99db8533e33 (table test-table [id=9302a2d0a69d436c9ffd7c4cd3562b1e]) successfully deleted
I20260620 04:25:11.719372 2722 ts_tablet_manager.cc:1918] T 4745b4addcc54013b27ff99db8533e33 P 3d79f1af503743878943a0d3ac3fe630: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 04:25:11.719431 2722 log.cc:1196] T 4745b4addcc54013b27ff99db8533e33 P 3d79f1af503743878943a0d3ac3fe630: Deleting WAL directory at /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929496330398-392-0/minicluster-data/ts-0/wal/wals/4745b4addcc54013b27ff99db8533e33
I20260620 04:25:11.719626 2722 ts_tablet_manager.cc:1939] T 4745b4addcc54013b27ff99db8533e33 P 3d79f1af503743878943a0d3ac3fe630: Deleting consensus metadata
I20260620 04:25:11.719825 2724 ts_tablet_manager.cc:1918] T 4745b4addcc54013b27ff99db8533e33 P d97a29850cc548d18b882742057e11dd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 04:25:11.719888 2724 log.cc:1196] T 4745b4addcc54013b27ff99db8533e33 P d97a29850cc548d18b882742057e11dd: Deleting WAL directory at /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929496330398-392-0/minicluster-data/ts-2/wal/wals/4745b4addcc54013b27ff99db8533e33
I20260620 04:25:11.720134 2724 ts_tablet_manager.cc:1939] T 4745b4addcc54013b27ff99db8533e33 P d97a29850cc548d18b882742057e11dd: Deleting consensus metadata
I20260620 04:25:11.720252 2115 catalog_manager.cc:4841] TS 3d79f1af503743878943a0d3ac3fe630 (127.0.98.1:37973): tablet 4745b4addcc54013b27ff99db8533e33 (table test-table [id=9302a2d0a69d436c9ffd7c4cd3562b1e]) successfully deleted
I20260620 04:25:11.720721 2117 catalog_manager.cc:4841] TS d97a29850cc548d18b882742057e11dd (127.0.98.3:45271): tablet 4745b4addcc54013b27ff99db8533e33 (table test-table [id=9302a2d0a69d436c9ffd7c4cd3562b1e]) successfully deleted
Jun 20 04:25:11 dist-test-slave-nk6d krb5kdc[2085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929511, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 04:25:11 dist-test-slave-nk6d krb5kdc[2085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929511, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:25:11.741531 2732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:11.733449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38092 (local address 127.0.98.62:34081)
0620 04:25:11.733695 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:11.733704 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:11.733722 (+ 18us) server_negotiation.cc:407] Connection header received
0620 04:25:11.733826 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:11.733829 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:11.733879 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:11.733967 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:11.734710 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:11.735256 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:11.736194 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:11.736472 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:11.736555 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:11.738723 (+ 2168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:11.738726 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:11.738728 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:25:11.738751 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:11.740118 (+ 1367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:11.740518 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:11.740521 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:11.740522 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:11.740588 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:11.740841 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:11.740843 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:11.740844 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:11.740998 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:11.741090 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:11.741301 (+ 211us) server_negotiation.cc:299] Negotiation successful
0620 04:25:11.741398 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":82,"threads_started":1}
I20260620 04:25:11.743785 2129 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38092:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 04:25:11.744213 2129 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 04:25:11.748445 2267 tablet_service.cc:1467] Processing CreateTablet for tablet 8c583d5a6cf64fa4badcb1aa08402b7a (DEFAULT_TABLE table=test-table [id=253efc3cd0b04d93bada98a6b1f04a94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:25:11.748545 2407 tablet_service.cc:1467] Processing CreateTablet for tablet 8c583d5a6cf64fa4badcb1aa08402b7a (DEFAULT_TABLE table=test-table [id=253efc3cd0b04d93bada98a6b1f04a94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:25:11.748768 2267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c583d5a6cf64fa4badcb1aa08402b7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:25:11.748806 2407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c583d5a6cf64fa4badcb1aa08402b7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:25:11.748829 2547 tablet_service.cc:1467] Processing CreateTablet for tablet 8c583d5a6cf64fa4badcb1aa08402b7a (DEFAULT_TABLE table=test-table [id=253efc3cd0b04d93bada98a6b1f04a94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:25:11.749092 2547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c583d5a6cf64fa4badcb1aa08402b7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:25:11.750746 2733 tablet_bootstrap.cc:492] T 8c583d5a6cf64fa4badcb1aa08402b7a P d4cfed1c9d714d02a3797ec591c46bed: Bootstrap starting.
I20260620 04:25:11.751200 2734 tablet_bootstrap.cc:492] T 8c583d5a6cf64fa4badcb1aa08402b7a P 3d79f1af503743878943a0d3ac3fe630: Bootstrap starting.
I20260620 04:25:11.751502 2735 tablet_bootstrap.cc:492] T 8c583d5a6cf64fa4badcb1aa08402b7a P d97a29850cc548d18b882742057e11dd: Bootstrap starting.
I20260620 04:25:11.751717 2733 tablet_bootstrap.cc:654] T 8c583d5a6cf64fa4badcb1aa08402b7a P d4cfed1c9d714d02a3797ec591c46bed: Neither blocks nor log segments found. Creating new log.
I20260620 04:25:11.752213 2734 tablet_bootstrap.cc:654] T 8c583d5a6cf64fa4badcb1aa08402b7a P 3d79f1af503743878943a0d3ac3fe630: Neither blocks nor log segments found. Creating new log.
I20260620 04:25:11.752346 2735 tablet_bootstrap.cc:654] T 8c583d5a6cf64fa4badcb1aa08402b7a P d97a29850cc548d18b882742057e11dd: Neither blocks nor log segments found. Creating new log.
I20260620 04:25:11.753170 2733 tablet_bootstrap.cc:492] T 8c583d5a6cf64fa4badcb1aa08402b7a P d4cfed1c9d714d02a3797ec591c46bed: No bootstrap required, opened a new log
I20260620 04:25:11.753222 2734 tablet_bootstrap.cc:492] T 8c583d5a6cf64fa4badcb1aa08402b7a P 3d79f1af503743878943a0d3ac3fe630: No bootstrap required, opened a new log
I20260620 04:25:11.753255 2733 ts_tablet_manager.cc:1397] T 8c583d5a6cf64fa4badcb1aa08402b7a P d4cfed1c9d714d02a3797ec591c46bed: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260620 04:25:11.753295 2734 ts_tablet_manager.cc:1397] T 8c583d5a6cf64fa4badcb1aa08402b7a P 3d79f1af503743878943a0d3ac3fe630: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 04:25:11.753422 2735 tablet_bootstrap.cc:492] T 8c583d5a6cf64fa4badcb1aa08402b7a P d97a29850cc548d18b882742057e11dd: No bootstrap required, opened a new log
I20260620 04:25:11.753501 2735 ts_tablet_manager.cc:1397] T 8c583d5a6cf64fa4badcb1aa08402b7a P d97a29850cc548d18b882742057e11dd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 04:25:11.753655 2733 raft_consensus.cc:348] T 8c583d5a6cf64fa4badcb1aa08402b7a P d4cfed1c9d714d02a3797ec591c46bed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d79f1af503743878943a0d3ac3fe630" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "d4cfed1c9d714d02a3797ec591c46bed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38267 } } peers { permanent_uuid: "d97a29850cc548d18b882742057e11dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45271 } }
I20260620 04:25:11.753693 2734 raft_consensus.cc:348] T 8c583d5a6cf64fa4badcb1aa08402b7a P 3d79f1af503743878943a0d3ac3fe630 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d79f1af503743878943a0d3ac3fe630" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "d4cfed1c9d714d02a3797ec591c46bed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38267 } } peers { permanent_uuid: "d97a29850cc548d18b882742057e11dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45271 } }
I20260620 04:25:11.753783 2733 raft_consensus.cc:374] T 8c583d5a6cf64fa4badcb1aa08402b7a P d4cfed1c9d714d02a3797ec591c46bed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:25:11.753789 2734 raft_consensus.cc:374] T 8c583d5a6cf64fa4badcb1aa08402b7a P 3d79f1af503743878943a0d3ac3fe630 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:25:11.753823 2733 raft_consensus.cc:729] T 8c583d5a6cf64fa4badcb1aa08402b7a P d4cfed1c9d714d02a3797ec591c46bed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4cfed1c9d714d02a3797ec591c46bed, State: Initialized, Role: FOLLOWER
I20260620 04:25:11.753847 2735 raft_consensus.cc:348] T 8c583d5a6cf64fa4badcb1aa08402b7a P d97a29850cc548d18b882742057e11dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d79f1af503743878943a0d3ac3fe630" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "d4cfed1c9d714d02a3797ec591c46bed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38267 } } peers { permanent_uuid: "d97a29850cc548d18b882742057e11dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45271 } }
I20260620 04:25:11.753935 2735 raft_consensus.cc:374] T 8c583d5a6cf64fa4badcb1aa08402b7a P d97a29850cc548d18b882742057e11dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:25:11.753969 2735 raft_consensus.cc:729] T 8c583d5a6cf64fa4badcb1aa08402b7a P d97a29850cc548d18b882742057e11dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d97a29850cc548d18b882742057e11dd, State: Initialized, Role: FOLLOWER
I20260620 04:25:11.753944 2733 consensus_queue.cc:260] T 8c583d5a6cf64fa4badcb1aa08402b7a P d4cfed1c9d714d02a3797ec591c46bed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d79f1af503743878943a0d3ac3fe630" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "d4cfed1c9d714d02a3797ec591c46bed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38267 } } peers { permanent_uuid: "d97a29850cc548d18b882742057e11dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45271 } }
I20260620 04:25:11.754112 2735 consensus_queue.cc:260] T 8c583d5a6cf64fa4badcb1aa08402b7a P d97a29850cc548d18b882742057e11dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d79f1af503743878943a0d3ac3fe630" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "d4cfed1c9d714d02a3797ec591c46bed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38267 } } peers { permanent_uuid: "d97a29850cc548d18b882742057e11dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45271 } }
I20260620 04:25:11.753822 2734 raft_consensus.cc:729] T 8c583d5a6cf64fa4badcb1aa08402b7a P 3d79f1af503743878943a0d3ac3fe630 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d79f1af503743878943a0d3ac3fe630, State: Initialized, Role: FOLLOWER
I20260620 04:25:11.754283 2733 ts_tablet_manager.cc:1428] T 8c583d5a6cf64fa4badcb1aa08402b7a P d4cfed1c9d714d02a3797ec591c46bed: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 04:25:11.754356 2735 ts_tablet_manager.cc:1428] T 8c583d5a6cf64fa4badcb1aa08402b7a P d97a29850cc548d18b882742057e11dd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 04:25:11.754338 2734 consensus_queue.cc:260] T 8c583d5a6cf64fa4badcb1aa08402b7a P 3d79f1af503743878943a0d3ac3fe630 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d79f1af503743878943a0d3ac3fe630" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "d4cfed1c9d714d02a3797ec591c46bed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38267 } } peers { permanent_uuid: "d97a29850cc548d18b882742057e11dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45271 } }
I20260620 04:25:11.754588 2734 ts_tablet_manager.cc:1428] T 8c583d5a6cf64fa4badcb1aa08402b7a P 3d79f1af503743878943a0d3ac3fe630: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 04:25:12.002012 2716 raft_consensus.cc:482] T 8c583d5a6cf64fa4badcb1aa08402b7a P d97a29850cc548d18b882742057e11dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:25:12.002167 2716 raft_consensus.cc:504] T 8c583d5a6cf64fa4badcb1aa08402b7a P d97a29850cc548d18b882742057e11dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d79f1af503743878943a0d3ac3fe630" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "d4cfed1c9d714d02a3797ec591c46bed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38267 } } peers { permanent_uuid: "d97a29850cc548d18b882742057e11dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45271 } }
I20260620 04:25:12.002635 2716 leader_election.cc:288] T 8c583d5a6cf64fa4badcb1aa08402b7a P d97a29850cc548d18b882742057e11dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d79f1af503743878943a0d3ac3fe630 (127.0.98.1:37973), d4cfed1c9d714d02a3797ec591c46bed (127.0.98.2:38267)
I20260620 04:25:12.003191 2426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c583d5a6cf64fa4badcb1aa08402b7a" candidate_uuid: "d97a29850cc548d18b882742057e11dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4cfed1c9d714d02a3797ec591c46bed" is_pre_election: true
I20260620 04:25:12.003189 2287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c583d5a6cf64fa4badcb1aa08402b7a" candidate_uuid: "d97a29850cc548d18b882742057e11dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d79f1af503743878943a0d3ac3fe630" is_pre_election: true
I20260620 04:25:12.003347 2287 raft_consensus.cc:2454] T 8c583d5a6cf64fa4badcb1aa08402b7a P 3d79f1af503743878943a0d3ac3fe630 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d97a29850cc548d18b882742057e11dd in term 0.
I20260620 04:25:12.003347 2426 raft_consensus.cc:2454] T 8c583d5a6cf64fa4badcb1aa08402b7a P d4cfed1c9d714d02a3797ec591c46bed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d97a29850cc548d18b882742057e11dd in term 0.
I20260620 04:25:12.003718 2499 leader_election.cc:302] T 8c583d5a6cf64fa4badcb1aa08402b7a P d97a29850cc548d18b882742057e11dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3d79f1af503743878943a0d3ac3fe630, d4cfed1c9d714d02a3797ec591c46bed, d97a29850cc548d18b882742057e11dd; no voters:
I20260620 04:25:12.003916 2716 raft_consensus.cc:2781] T 8c583d5a6cf64fa4badcb1aa08402b7a P d97a29850cc548d18b882742057e11dd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 04:25:12.004057 2716 raft_consensus.cc:482] T 8c583d5a6cf64fa4badcb1aa08402b7a P d97a29850cc548d18b882742057e11dd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 04:25:12.004110 2716 raft_consensus.cc:3037] T 8c583d5a6cf64fa4badcb1aa08402b7a P d97a29850cc548d18b882742057e11dd [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:25:12.005178 2716 raft_consensus.cc:504] T 8c583d5a6cf64fa4badcb1aa08402b7a P d97a29850cc548d18b882742057e11dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d79f1af503743878943a0d3ac3fe630" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "d4cfed1c9d714d02a3797ec591c46bed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38267 } } peers { permanent_uuid: "d97a29850cc548d18b882742057e11dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45271 } }
I20260620 04:25:12.005523 2716 leader_election.cc:288] T 8c583d5a6cf64fa4badcb1aa08402b7a P d97a29850cc548d18b882742057e11dd [CANDIDATE]: Term 1 election: Requested vote from peers 3d79f1af503743878943a0d3ac3fe630 (127.0.98.1:37973), d4cfed1c9d714d02a3797ec591c46bed (127.0.98.2:38267)
I20260620 04:25:12.005873 2287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c583d5a6cf64fa4badcb1aa08402b7a" candidate_uuid: "d97a29850cc548d18b882742057e11dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d79f1af503743878943a0d3ac3fe630"
I20260620 04:25:12.005901 2426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c583d5a6cf64fa4badcb1aa08402b7a" candidate_uuid: "d97a29850cc548d18b882742057e11dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4cfed1c9d714d02a3797ec591c46bed"
I20260620 04:25:12.005988 2287 raft_consensus.cc:3037] T 8c583d5a6cf64fa4badcb1aa08402b7a P 3d79f1af503743878943a0d3ac3fe630 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:25:12.006016 2426 raft_consensus.cc:3037] T 8c583d5a6cf64fa4badcb1aa08402b7a P d4cfed1c9d714d02a3797ec591c46bed [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:25:12.006886 2287 raft_consensus.cc:2454] T 8c583d5a6cf64fa4badcb1aa08402b7a P 3d79f1af503743878943a0d3ac3fe630 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d97a29850cc548d18b882742057e11dd in term 1.
I20260620 04:25:12.006925 2426 raft_consensus.cc:2454] T 8c583d5a6cf64fa4badcb1aa08402b7a P d4cfed1c9d714d02a3797ec591c46bed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d97a29850cc548d18b882742057e11dd in term 1.
I20260620 04:25:12.007195 2499 leader_election.cc:302] T 8c583d5a6cf64fa4badcb1aa08402b7a P d97a29850cc548d18b882742057e11dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d79f1af503743878943a0d3ac3fe630, d97a29850cc548d18b882742057e11dd; no voters:
I20260620 04:25:12.007370 2716 raft_consensus.cc:2781] T 8c583d5a6cf64fa4badcb1aa08402b7a P d97a29850cc548d18b882742057e11dd [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:25:12.007452 2716 raft_consensus.cc:686] T 8c583d5a6cf64fa4badcb1aa08402b7a P d97a29850cc548d18b882742057e11dd [term 1 LEADER]: Becoming Leader. State: Replica: d97a29850cc548d18b882742057e11dd, State: Running, Role: LEADER
I20260620 04:25:12.007579 2716 consensus_queue.cc:237] T 8c583d5a6cf64fa4badcb1aa08402b7a P d97a29850cc548d18b882742057e11dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d79f1af503743878943a0d3ac3fe630" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "d4cfed1c9d714d02a3797ec591c46bed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38267 } } peers { permanent_uuid: "d97a29850cc548d18b882742057e11dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45271 } }
I20260620 04:25:12.009338 2128 catalog_manager.cc:5495] T 8c583d5a6cf64fa4badcb1aa08402b7a P d97a29850cc548d18b882742057e11dd reported cstate change: term changed from 0 to 1, leader changed from <none> to d97a29850cc548d18b882742057e11dd (127.0.98.3). New cstate: current_term: 1 leader_uuid: "d97a29850cc548d18b882742057e11dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d79f1af503743878943a0d3ac3fe630" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4cfed1c9d714d02a3797ec591c46bed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d97a29850cc548d18b882742057e11dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45271 } health_report { overall_health: HEALTHY } } }
I20260620 04:25:12.053812 2742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:12.050450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56482 (local address 127.0.98.3:45271)
0620 04:25:12.050713 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:12.050719 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:12.050732 (+ 13us) server_negotiation.cc:407] Connection header received
0620 04:25:12.050768 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:12.050771 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:12.050810 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:12.050876 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:25:12.051224 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:12.051792 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:12.052716 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:12.052963 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:12.053059 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:12.053327 (+ 268us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:25:12.053395 (+ 68us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:25:12.053445 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:12.053613 (+ 168us) server_negotiation.cc:299] Negotiation successful
0620 04:25:12.053665 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":107,"threads_started":1}
I20260620 04:25:12.060177 2426 raft_consensus.cc:1261] T 8c583d5a6cf64fa4badcb1aa08402b7a P d4cfed1c9d714d02a3797ec591c46bed [term 1 FOLLOWER]: Refusing update from remote peer d97a29850cc548d18b882742057e11dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:25:12.060189 2287 raft_consensus.cc:1261] T 8c583d5a6cf64fa4badcb1aa08402b7a P 3d79f1af503743878943a0d3ac3fe630 [term 1 FOLLOWER]: Refusing update from remote peer d97a29850cc548d18b882742057e11dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:25:12.060535 2713 consensus_queue.cc:1035] T 8c583d5a6cf64fa4badcb1aa08402b7a P d97a29850cc548d18b882742057e11dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4cfed1c9d714d02a3797ec591c46bed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:25:12.060691 2716 consensus_queue.cc:1035] T 8c583d5a6cf64fa4badcb1aa08402b7a P d97a29850cc548d18b882742057e11dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d79f1af503743878943a0d3ac3fe630" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:25:12.087466 2130 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38092:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:25:12.087637 2130 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38092:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:25:12.090015 2130 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 15ec2623f54b4519af25addf17f1c427: Sending DeleteTablet for 3 replicas of tablet 8c583d5a6cf64fa4badcb1aa08402b7a
I20260620 04:25:12.090631 2407 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c583d5a6cf64fa4badcb1aa08402b7a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:25:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34240
I20260620 04:25:12.090636 2267 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c583d5a6cf64fa4badcb1aa08402b7a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:25:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48964
I20260620 04:25:12.090796 2722 tablet_replica.cc:330] T 8c583d5a6cf64fa4badcb1aa08402b7a P 3d79f1af503743878943a0d3ac3fe630: stopping tablet replica
I20260620 04:25:12.090796 2723 tablet_replica.cc:330] T 8c583d5a6cf64fa4badcb1aa08402b7a P d4cfed1c9d714d02a3797ec591c46bed: stopping tablet replica
I20260620 04:25:12.090857 2547 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c583d5a6cf64fa4badcb1aa08402b7a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:25:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34728
I20260620 04:25:12.090899 2723 raft_consensus.cc:2229] T 8c583d5a6cf64fa4badcb1aa08402b7a P d4cfed1c9d714d02a3797ec591c46bed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:25:12.090899 2722 raft_consensus.cc:2229] T 8c583d5a6cf64fa4badcb1aa08402b7a P 3d79f1af503743878943a0d3ac3fe630 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:25:12.090996 2722 raft_consensus.cc:2258] T 8c583d5a6cf64fa4badcb1aa08402b7a P 3d79f1af503743878943a0d3ac3fe630 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:25:12.090996 2723 raft_consensus.cc:2258] T 8c583d5a6cf64fa4badcb1aa08402b7a P d4cfed1c9d714d02a3797ec591c46bed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:25:12.091158 2724 tablet_replica.cc:330] T 8c583d5a6cf64fa4badcb1aa08402b7a P d97a29850cc548d18b882742057e11dd: stopping tablet replica
I20260620 04:25:12.091265 2724 raft_consensus.cc:2229] T 8c583d5a6cf64fa4badcb1aa08402b7a P d97a29850cc548d18b882742057e11dd [term 1 LEADER]: Raft consensus shutting down.
I20260620 04:25:12.091449 2724 raft_consensus.cc:2258] T 8c583d5a6cf64fa4badcb1aa08402b7a P d97a29850cc548d18b882742057e11dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:25:12.091666 2723 ts_tablet_manager.cc:1905] T 8c583d5a6cf64fa4badcb1aa08402b7a P d4cfed1c9d714d02a3797ec591c46bed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:25:12.092588 2724 ts_tablet_manager.cc:1905] T 8c583d5a6cf64fa4badcb1aa08402b7a P d97a29850cc548d18b882742057e11dd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:25:12.093575 2130 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38092:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 04:25:12.094198 2130 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 04:25:12.094246 2723 ts_tablet_manager.cc:1918] T 8c583d5a6cf64fa4badcb1aa08402b7a P d4cfed1c9d714d02a3797ec591c46bed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 04:25:12.094319 2723 log.cc:1196] T 8c583d5a6cf64fa4badcb1aa08402b7a P d4cfed1c9d714d02a3797ec591c46bed: Deleting WAL directory at /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929496330398-392-0/minicluster-data/ts-1/wal/wals/8c583d5a6cf64fa4badcb1aa08402b7a
I20260620 04:25:12.094558 2723 ts_tablet_manager.cc:1939] T 8c583d5a6cf64fa4badcb1aa08402b7a P d4cfed1c9d714d02a3797ec591c46bed: Deleting consensus metadata
I20260620 04:25:12.095558 2724 ts_tablet_manager.cc:1918] T 8c583d5a6cf64fa4badcb1aa08402b7a P d97a29850cc548d18b882742057e11dd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 04:25:12.095634 2724 log.cc:1196] T 8c583d5a6cf64fa4badcb1aa08402b7a P d97a29850cc548d18b882742057e11dd: Deleting WAL directory at /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929496330398-392-0/minicluster-data/ts-2/wal/wals/8c583d5a6cf64fa4badcb1aa08402b7a
I20260620 04:25:12.095870 2724 ts_tablet_manager.cc:1939] T 8c583d5a6cf64fa4badcb1aa08402b7a P d97a29850cc548d18b882742057e11dd: Deleting consensus metadata
I20260620 04:25:12.096418 2116 catalog_manager.cc:4841] TS d4cfed1c9d714d02a3797ec591c46bed (127.0.98.2:38267): tablet 8c583d5a6cf64fa4badcb1aa08402b7a (table test-table [id=253efc3cd0b04d93bada98a6b1f04a94]) successfully deleted
I20260620 04:25:12.096774 2722 ts_tablet_manager.cc:1905] T 8c583d5a6cf64fa4badcb1aa08402b7a P 3d79f1af503743878943a0d3ac3fe630: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:25:12.099171 2722 ts_tablet_manager.cc:1918] T 8c583d5a6cf64fa4badcb1aa08402b7a P 3d79f1af503743878943a0d3ac3fe630: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 04:25:12.099241 2722 log.cc:1196] T 8c583d5a6cf64fa4badcb1aa08402b7a P 3d79f1af503743878943a0d3ac3fe630: Deleting WAL directory at /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929496330398-392-0/minicluster-data/ts-0/wal/wals/8c583d5a6cf64fa4badcb1aa08402b7a
I20260620 04:25:12.099435 2722 ts_tablet_manager.cc:1939] T 8c583d5a6cf64fa4badcb1aa08402b7a P 3d79f1af503743878943a0d3ac3fe630: Deleting consensus metadata
I20260620 04:25:12.100416 2115 catalog_manager.cc:4841] TS 3d79f1af503743878943a0d3ac3fe630 (127.0.98.1:37973): tablet 8c583d5a6cf64fa4badcb1aa08402b7a (table test-table [id=253efc3cd0b04d93bada98a6b1f04a94]) successfully deleted
I20260620 04:25:12.101197 2117 catalog_manager.cc:4841] TS d97a29850cc548d18b882742057e11dd (127.0.98.3:45271): tablet 8c583d5a6cf64fa4badcb1aa08402b7a (table test-table [id=253efc3cd0b04d93bada98a6b1f04a94]) successfully deleted
I20260620 04:25:12.101737 2407 tablet_service.cc:1467] Processing CreateTablet for tablet 601482ab8ead4ad5907b77bceb75e4d0 (DEFAULT_TABLE table=test-table [id=2dfb057585c5452596b255b283d8802f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:25:12.101737 2267 tablet_service.cc:1467] Processing CreateTablet for tablet 601482ab8ead4ad5907b77bceb75e4d0 (DEFAULT_TABLE table=test-table [id=2dfb057585c5452596b255b283d8802f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:25:12.101768 2547 tablet_service.cc:1467] Processing CreateTablet for tablet 601482ab8ead4ad5907b77bceb75e4d0 (DEFAULT_TABLE table=test-table [id=2dfb057585c5452596b255b283d8802f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:25:12.101974 2267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 601482ab8ead4ad5907b77bceb75e4d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:25:12.101974 2407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 601482ab8ead4ad5907b77bceb75e4d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:25:12.102015 2547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 601482ab8ead4ad5907b77bceb75e4d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:25:12.103946 2735 tablet_bootstrap.cc:492] T 601482ab8ead4ad5907b77bceb75e4d0 P d97a29850cc548d18b882742057e11dd: Bootstrap starting.
I20260620 04:25:12.104174 2733 tablet_bootstrap.cc:492] T 601482ab8ead4ad5907b77bceb75e4d0 P d4cfed1c9d714d02a3797ec591c46bed: Bootstrap starting.
I20260620 04:25:12.104173 2734 tablet_bootstrap.cc:492] T 601482ab8ead4ad5907b77bceb75e4d0 P 3d79f1af503743878943a0d3ac3fe630: Bootstrap starting.
I20260620 04:25:12.104775 2735 tablet_bootstrap.cc:654] T 601482ab8ead4ad5907b77bceb75e4d0 P d97a29850cc548d18b882742057e11dd: Neither blocks nor log segments found. Creating new log.
I20260620 04:25:12.104878 2734 tablet_bootstrap.cc:654] T 601482ab8ead4ad5907b77bceb75e4d0 P 3d79f1af503743878943a0d3ac3fe630: Neither blocks nor log segments found. Creating new log.
I20260620 04:25:12.104907 2733 tablet_bootstrap.cc:654] T 601482ab8ead4ad5907b77bceb75e4d0 P d4cfed1c9d714d02a3797ec591c46bed: Neither blocks nor log segments found. Creating new log.
I20260620 04:25:12.105906 2735 tablet_bootstrap.cc:492] T 601482ab8ead4ad5907b77bceb75e4d0 P d97a29850cc548d18b882742057e11dd: No bootstrap required, opened a new log
I20260620 04:25:12.105993 2735 ts_tablet_manager.cc:1397] T 601482ab8ead4ad5907b77bceb75e4d0 P d97a29850cc548d18b882742057e11dd: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 04:25:12.106382 2735 raft_consensus.cc:348] T 601482ab8ead4ad5907b77bceb75e4d0 P d97a29850cc548d18b882742057e11dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d79f1af503743878943a0d3ac3fe630" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "d4cfed1c9d714d02a3797ec591c46bed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38267 } } peers { permanent_uuid: "d97a29850cc548d18b882742057e11dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45271 } }
I20260620 04:25:12.106479 2735 raft_consensus.cc:374] T 601482ab8ead4ad5907b77bceb75e4d0 P d97a29850cc548d18b882742057e11dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:25:12.106500 2735 raft_consensus.cc:729] T 601482ab8ead4ad5907b77bceb75e4d0 P d97a29850cc548d18b882742057e11dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d97a29850cc548d18b882742057e11dd, State: Initialized, Role: FOLLOWER
I20260620 04:25:12.106596 2735 consensus_queue.cc:260] T 601482ab8ead4ad5907b77bceb75e4d0 P d97a29850cc548d18b882742057e11dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d79f1af503743878943a0d3ac3fe630" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "d4cfed1c9d714d02a3797ec591c46bed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38267 } } peers { permanent_uuid: "d97a29850cc548d18b882742057e11dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45271 } }
I20260620 04:25:12.106843 2735 ts_tablet_manager.cc:1428] T 601482ab8ead4ad5907b77bceb75e4d0 P d97a29850cc548d18b882742057e11dd: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 04:25:12.107355 2734 tablet_bootstrap.cc:492] T 601482ab8ead4ad5907b77bceb75e4d0 P 3d79f1af503743878943a0d3ac3fe630: No bootstrap required, opened a new log
I20260620 04:25:12.107441 2734 ts_tablet_manager.cc:1397] T 601482ab8ead4ad5907b77bceb75e4d0 P 3d79f1af503743878943a0d3ac3fe630: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 04:25:12.107656 2733 tablet_bootstrap.cc:492] T 601482ab8ead4ad5907b77bceb75e4d0 P d4cfed1c9d714d02a3797ec591c46bed: No bootstrap required, opened a new log
I20260620 04:25:12.107728 2733 ts_tablet_manager.cc:1397] T 601482ab8ead4ad5907b77bceb75e4d0 P d4cfed1c9d714d02a3797ec591c46bed: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260620 04:25:12.107785 2734 raft_consensus.cc:348] T 601482ab8ead4ad5907b77bceb75e4d0 P 3d79f1af503743878943a0d3ac3fe630 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d79f1af503743878943a0d3ac3fe630" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "d4cfed1c9d714d02a3797ec591c46bed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38267 } } peers { permanent_uuid: "d97a29850cc548d18b882742057e11dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45271 } }
I20260620 04:25:12.107870 2734 raft_consensus.cc:374] T 601482ab8ead4ad5907b77bceb75e4d0 P 3d79f1af503743878943a0d3ac3fe630 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:25:12.107890 2734 raft_consensus.cc:729] T 601482ab8ead4ad5907b77bceb75e4d0 P 3d79f1af503743878943a0d3ac3fe630 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d79f1af503743878943a0d3ac3fe630, State: Initialized, Role: FOLLOWER
I20260620 04:25:12.108021 2734 consensus_queue.cc:260] T 601482ab8ead4ad5907b77bceb75e4d0 P 3d79f1af503743878943a0d3ac3fe630 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d79f1af503743878943a0d3ac3fe630" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "d4cfed1c9d714d02a3797ec591c46bed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38267 } } peers { permanent_uuid: "d97a29850cc548d18b882742057e11dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45271 } }
I20260620 04:25:12.108057 2733 raft_consensus.cc:348] T 601482ab8ead4ad5907b77bceb75e4d0 P d4cfed1c9d714d02a3797ec591c46bed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d79f1af503743878943a0d3ac3fe630" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "d4cfed1c9d714d02a3797ec591c46bed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38267 } } peers { permanent_uuid: "d97a29850cc548d18b882742057e11dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45271 } }
I20260620 04:25:12.108165 2733 raft_consensus.cc:374] T 601482ab8ead4ad5907b77bceb75e4d0 P d4cfed1c9d714d02a3797ec591c46bed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:25:12.108196 2733 raft_consensus.cc:729] T 601482ab8ead4ad5907b77bceb75e4d0 P d4cfed1c9d714d02a3797ec591c46bed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4cfed1c9d714d02a3797ec591c46bed, State: Initialized, Role: FOLLOWER
I20260620 04:25:12.108273 2734 ts_tablet_manager.cc:1428] T 601482ab8ead4ad5907b77bceb75e4d0 P 3d79f1af503743878943a0d3ac3fe630: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 04:25:12.108304 2733 consensus_queue.cc:260] T 601482ab8ead4ad5907b77bceb75e4d0 P d4cfed1c9d714d02a3797ec591c46bed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d79f1af503743878943a0d3ac3fe630" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "d4cfed1c9d714d02a3797ec591c46bed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38267 } } peers { permanent_uuid: "d97a29850cc548d18b882742057e11dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45271 } }
I20260620 04:25:12.108546 2733 ts_tablet_manager.cc:1428] T 601482ab8ead4ad5907b77bceb75e4d0 P d4cfed1c9d714d02a3797ec591c46bed: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 04:25:12.160584 2703 raft_consensus.cc:482] T 601482ab8ead4ad5907b77bceb75e4d0 P 3d79f1af503743878943a0d3ac3fe630 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:25:12.160744 2703 raft_consensus.cc:504] T 601482ab8ead4ad5907b77bceb75e4d0 P 3d79f1af503743878943a0d3ac3fe630 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d79f1af503743878943a0d3ac3fe630" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "d4cfed1c9d714d02a3797ec591c46bed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38267 } } peers { permanent_uuid: "d97a29850cc548d18b882742057e11dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45271 } }
I20260620 04:25:12.161129 2703 leader_election.cc:288] T 601482ab8ead4ad5907b77bceb75e4d0 P 3d79f1af503743878943a0d3ac3fe630 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4cfed1c9d714d02a3797ec591c46bed (127.0.98.2:38267), d97a29850cc548d18b882742057e11dd (127.0.98.3:45271)
I20260620 04:25:12.161574 2567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "601482ab8ead4ad5907b77bceb75e4d0" candidate_uuid: "3d79f1af503743878943a0d3ac3fe630" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d97a29850cc548d18b882742057e11dd" is_pre_election: true
I20260620 04:25:12.161607 2426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "601482ab8ead4ad5907b77bceb75e4d0" candidate_uuid: "3d79f1af503743878943a0d3ac3fe630" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4cfed1c9d714d02a3797ec591c46bed" is_pre_election: true
I20260620 04:25:12.161729 2567 raft_consensus.cc:2454] T 601482ab8ead4ad5907b77bceb75e4d0 P d97a29850cc548d18b882742057e11dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d79f1af503743878943a0d3ac3fe630 in term 0.
I20260620 04:25:12.161737 2426 raft_consensus.cc:2454] T 601482ab8ead4ad5907b77bceb75e4d0 P d4cfed1c9d714d02a3797ec591c46bed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d79f1af503743878943a0d3ac3fe630 in term 0.
I20260620 04:25:12.162035 2220 leader_election.cc:302] T 601482ab8ead4ad5907b77bceb75e4d0 P 3d79f1af503743878943a0d3ac3fe630 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d79f1af503743878943a0d3ac3fe630, d4cfed1c9d714d02a3797ec591c46bed; no voters:
I20260620 04:25:12.162240 2703 raft_consensus.cc:2781] T 601482ab8ead4ad5907b77bceb75e4d0 P 3d79f1af503743878943a0d3ac3fe630 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 04:25:12.162308 2703 raft_consensus.cc:482] T 601482ab8ead4ad5907b77bceb75e4d0 P 3d79f1af503743878943a0d3ac3fe630 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 04:25:12.162353 2703 raft_consensus.cc:3037] T 601482ab8ead4ad5907b77bceb75e4d0 P 3d79f1af503743878943a0d3ac3fe630 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:25:12.163831 2703 raft_consensus.cc:504] T 601482ab8ead4ad5907b77bceb75e4d0 P 3d79f1af503743878943a0d3ac3fe630 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d79f1af503743878943a0d3ac3fe630" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "d4cfed1c9d714d02a3797ec591c46bed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38267 } } peers { permanent_uuid: "d97a29850cc548d18b882742057e11dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45271 } }
I20260620 04:25:12.164191 2703 leader_election.cc:288] T 601482ab8ead4ad5907b77bceb75e4d0 P 3d79f1af503743878943a0d3ac3fe630 [CANDIDATE]: Term 1 election: Requested vote from peers d4cfed1c9d714d02a3797ec591c46bed (127.0.98.2:38267), d97a29850cc548d18b882742057e11dd (127.0.98.3:45271)
I20260620 04:25:12.164542 2426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "601482ab8ead4ad5907b77bceb75e4d0" candidate_uuid: "3d79f1af503743878943a0d3ac3fe630" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4cfed1c9d714d02a3797ec591c46bed"
I20260620 04:25:12.164562 2567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "601482ab8ead4ad5907b77bceb75e4d0" candidate_uuid: "3d79f1af503743878943a0d3ac3fe630" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d97a29850cc548d18b882742057e11dd"
I20260620 04:25:12.164670 2567 raft_consensus.cc:3037] T 601482ab8ead4ad5907b77bceb75e4d0 P d97a29850cc548d18b882742057e11dd [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:25:12.164652 2426 raft_consensus.cc:3037] T 601482ab8ead4ad5907b77bceb75e4d0 P d4cfed1c9d714d02a3797ec591c46bed [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:25:12.165683 2567 raft_consensus.cc:2454] T 601482ab8ead4ad5907b77bceb75e4d0 P d97a29850cc548d18b882742057e11dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d79f1af503743878943a0d3ac3fe630 in term 1.
I20260620 04:25:12.165771 2426 raft_consensus.cc:2454] T 601482ab8ead4ad5907b77bceb75e4d0 P d4cfed1c9d714d02a3797ec591c46bed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d79f1af503743878943a0d3ac3fe630 in term 1.
I20260620 04:25:12.165959 2221 leader_election.cc:302] T 601482ab8ead4ad5907b77bceb75e4d0 P 3d79f1af503743878943a0d3ac3fe630 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d79f1af503743878943a0d3ac3fe630, d97a29850cc548d18b882742057e11dd; no voters:
I20260620 04:25:12.166170 2703 raft_consensus.cc:2781] T 601482ab8ead4ad5907b77bceb75e4d0 P 3d79f1af503743878943a0d3ac3fe630 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:25:12.166251 2703 raft_consensus.cc:686] T 601482ab8ead4ad5907b77bceb75e4d0 P 3d79f1af503743878943a0d3ac3fe630 [term 1 LEADER]: Becoming Leader. State: Replica: 3d79f1af503743878943a0d3ac3fe630, State: Running, Role: LEADER
I20260620 04:25:12.166380 2703 consensus_queue.cc:237] T 601482ab8ead4ad5907b77bceb75e4d0 P 3d79f1af503743878943a0d3ac3fe630 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d79f1af503743878943a0d3ac3fe630" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "d4cfed1c9d714d02a3797ec591c46bed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38267 } } peers { permanent_uuid: "d97a29850cc548d18b882742057e11dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45271 } }
I20260620 04:25:12.167889 2130 catalog_manager.cc:5495] T 601482ab8ead4ad5907b77bceb75e4d0 P 3d79f1af503743878943a0d3ac3fe630 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d79f1af503743878943a0d3ac3fe630 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "3d79f1af503743878943a0d3ac3fe630" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d79f1af503743878943a0d3ac3fe630" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4cfed1c9d714d02a3797ec591c46bed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d97a29850cc548d18b882742057e11dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45271 } health_report { overall_health: UNKNOWN } } }
I20260620 04:25:12.194078 2750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:12.190662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41522 (local address 127.0.98.1:37973)
0620 04:25:12.191012 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:12.191018 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:12.191035 (+ 17us) server_negotiation.cc:407] Connection header received
0620 04:25:12.191068 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:12.191070 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:12.191108 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:12.191183 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:25:12.191424 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:12.191999 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:12.193144 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:12.193375 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:12.193449 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:12.193705 (+ 256us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:25:12.193777 (+ 72us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:25:12.193834 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:12.193906 (+ 72us) server_negotiation.cc:299] Negotiation successful
0620 04:25:12.193948 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":176,"threads_started":1}
I20260620 04:25:12.202510 2753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:12.199586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37973 (local address 127.0.98.1:49637)
0620 04:25:12.199846 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:12.199860 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:12.199987 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:12.200183 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:12.200185 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:12.200192 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 04:25:12.200298 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:12.200303 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:12.200961 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:12.200964 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:12.201534 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:12.201541 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:12.201922 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:12.201926 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:12.201997 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:12.202018 (+ 21us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 04:25:12.202240 (+ 222us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 04:25:12.202245 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 04:25:12.202296 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 04:25:12.202371 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"mutex_wait_us":2,"thread_start_us":89,"threads_started":1}
I20260620 04:25:12.202508 2750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:12.199723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49637 (local address 127.0.98.1:37973)
0620 04:25:12.199826 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:12.199830 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:12.199937 (+ 107us) server_negotiation.cc:407] Connection header received
0620 04:25:12.200073 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:12.200076 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:12.200115 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:12.200171 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:25:12.200380 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:12.200870 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:12.201629 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:12.201835 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:12.201915 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:12.202106 (+ 191us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:25:12.202169 (+ 63us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:25:12.202231 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:12.202314 (+ 83us) server_negotiation.cc:299] Negotiation successful
0620 04:25:12.202355 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 04:25:12.207662 2426 raft_consensus.cc:1261] T 601482ab8ead4ad5907b77bceb75e4d0 P d4cfed1c9d714d02a3797ec591c46bed [term 1 FOLLOWER]: Refusing update from remote peer 3d79f1af503743878943a0d3ac3fe630: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:25:12.207655 2567 raft_consensus.cc:1261] T 601482ab8ead4ad5907b77bceb75e4d0 P d97a29850cc548d18b882742057e11dd [term 1 FOLLOWER]: Refusing update from remote peer 3d79f1af503743878943a0d3ac3fe630: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:25:12.208063 2708 consensus_queue.cc:1035] T 601482ab8ead4ad5907b77bceb75e4d0 P 3d79f1af503743878943a0d3ac3fe630 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d97a29850cc548d18b882742057e11dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:25:12.208279 2659 consensus_queue.cc:1035] T 601482ab8ead4ad5907b77bceb75e4d0 P 3d79f1af503743878943a0d3ac3fe630 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4cfed1c9d714d02a3797ec591c46bed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 04:25:12.219043 2701 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 04:25:13.770130 2128 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38092:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:25:13.770288 2128 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38092:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:25:13.772686 2128 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 15ec2623f54b4519af25addf17f1c427: Sending DeleteTablet for 3 replicas of tablet 601482ab8ead4ad5907b77bceb75e4d0
I20260620 04:25:13.773234 2407 tablet_service.cc:1514] Processing DeleteTablet for tablet 601482ab8ead4ad5907b77bceb75e4d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:25:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34240
I20260620 04:25:13.773352 2267 tablet_service.cc:1514] Processing DeleteTablet for tablet 601482ab8ead4ad5907b77bceb75e4d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:25:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48964
I20260620 04:25:13.773599 2777 tablet_replica.cc:330] T 601482ab8ead4ad5907b77bceb75e4d0 P d4cfed1c9d714d02a3797ec591c46bed: stopping tablet replica
I20260620 04:25:13.773630 2547 tablet_service.cc:1514] Processing DeleteTablet for tablet 601482ab8ead4ad5907b77bceb75e4d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:25:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34728
I20260620 04:25:13.773677 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 2202
I20260620 04:25:13.773725 2777 raft_consensus.cc:2229] T 601482ab8ead4ad5907b77bceb75e4d0 P d4cfed1c9d714d02a3797ec591c46bed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:25:13.773846 2777 raft_consensus.cc:2258] T 601482ab8ead4ad5907b77bceb75e4d0 P d4cfed1c9d714d02a3797ec591c46bed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:25:13.774652 2777 ts_tablet_manager.cc:1905] T 601482ab8ead4ad5907b77bceb75e4d0 P d4cfed1c9d714d02a3797ec591c46bed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:25:13.775014 2779 tablet_replica.cc:330] T 601482ab8ead4ad5907b77bceb75e4d0 P d97a29850cc548d18b882742057e11dd: stopping tablet replica
I20260620 04:25:13.775127 2779 raft_consensus.cc:2229] T 601482ab8ead4ad5907b77bceb75e4d0 P d97a29850cc548d18b882742057e11dd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:25:13.775223 2779 raft_consensus.cc:2258] T 601482ab8ead4ad5907b77bceb75e4d0 P d97a29850cc548d18b882742057e11dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:25:13.775794 2779 ts_tablet_manager.cc:1905] T 601482ab8ead4ad5907b77bceb75e4d0 P d97a29850cc548d18b882742057e11dd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:25:13.777326 2777 ts_tablet_manager.cc:1918] T 601482ab8ead4ad5907b77bceb75e4d0 P d4cfed1c9d714d02a3797ec591c46bed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 04:25:13.777403 2777 log.cc:1196] T 601482ab8ead4ad5907b77bceb75e4d0 P d4cfed1c9d714d02a3797ec591c46bed: Deleting WAL directory at /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929496330398-392-0/minicluster-data/ts-1/wal/wals/601482ab8ead4ad5907b77bceb75e4d0
I20260620 04:25:13.777669 2777 ts_tablet_manager.cc:1939] T 601482ab8ead4ad5907b77bceb75e4d0 P d4cfed1c9d714d02a3797ec591c46bed: Deleting consensus metadata
I20260620 04:25:13.778298 2779 ts_tablet_manager.cc:1918] T 601482ab8ead4ad5907b77bceb75e4d0 P d97a29850cc548d18b882742057e11dd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 04:25:13.778368 2779 log.cc:1196] T 601482ab8ead4ad5907b77bceb75e4d0 P d97a29850cc548d18b882742057e11dd: Deleting WAL directory at /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929496330398-392-0/minicluster-data/ts-2/wal/wals/601482ab8ead4ad5907b77bceb75e4d0
I20260620 04:25:13.778366 2116 catalog_manager.cc:4841] TS d4cfed1c9d714d02a3797ec591c46bed (127.0.98.2:38267): tablet 601482ab8ead4ad5907b77bceb75e4d0 (table test-table [id=2dfb057585c5452596b255b283d8802f]) successfully deleted
I20260620 04:25:13.778594 2779 ts_tablet_manager.cc:1939] T 601482ab8ead4ad5907b77bceb75e4d0 P d97a29850cc548d18b882742057e11dd: Deleting consensus metadata
I20260620 04:25:13.779148 2117 catalog_manager.cc:4841] TS d97a29850cc548d18b882742057e11dd (127.0.98.3:45271): tablet 601482ab8ead4ad5907b77bceb75e4d0 (table test-table [id=2dfb057585c5452596b255b283d8802f]) successfully deleted
W20260620 04:25:13.781322 2115 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37973 (error 108)
I20260620 04:25:13.781481 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 2342
I20260620 04:25:13.783082 2781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:13.782325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37973 (local address 127.0.0.1:41534)
0620 04:25:13.782898 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:13.782980 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":68,"threads_started":1}
W20260620 04:25:13.783217 2115 catalog_manager.cc:4584] TS 3d79f1af503743878943a0d3ac3fe630 (127.0.98.1:37973): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 601482ab8ead4ad5907b77bceb75e4d0: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37973: connect: Connection refused (error 111)
I20260620 04:25:13.787707 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 2482
I20260620 04:25:13.793221 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 2098
2026-06-20T04:25:13Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6135 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929496330398-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:25:13 dist-test-slave-nk6d krb5kdc[2784](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:25:13 dist-test-slave-nk6d krb5kdc[2784](info): set up 2 sockets
Jun 20 04:25:13 dist-test-slave-nk6d krb5kdc[2784](info): commencing operation
krb5kdc: starting...
W20260620 04:25:14.743652 392 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929496330398-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929496330398-392-0/krb5kdc/test-user.keytab.
Jun 20 04:25:14 dist-test-slave-nk6d krb5kdc[2784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929514, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:25:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:25:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 04:25:14.856329 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929496330398-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929496330398-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929496330398-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39795
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39795
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:25:14.962414 2797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:25:14.962667 2797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:25:14.962705 2797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:25:14.966655 2797 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:25:14.966719 2797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:25:14.966737 2797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:25:14.966753 2797 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:25:14.966768 2797 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:25:14.972106 2797 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929496330398-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929496330398-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39795
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2797
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929496330398-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:25:14.973174 2797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:25:14.974213 2797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 04:25:14.982462 2797 server_base.cc:1034] running on GCE node
W20260620 04:25:14.982410 2803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:25:14.982470 2802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:25:14.982399 2805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:25:14.983263 2797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:25:14.984398 2797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:25:14.985567 2797 hybrid_clock.cc:648] HybridClock initialized: now 1781929514985547 us; error 32 us; skew 500 ppm
Jun 20 04:25:14 dist-test-slave-nk6d krb5kdc[2784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929514, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:25:14.994503 2797 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 04:25:14.995434 2797 webserver.cc:460] Webserver started at http://127.0.98.62:38763/ using document root <none> and password file <none>
I20260620 04:25:14.996044 2797 fs_manager.cc:359] Metadata directory not provided
I20260620 04:25:14.996116 2797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:25:14.996320 2797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:25:14.998383 2797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929496330398-392-0/minicluster-data/master-0/data/instance:
uuid: "f00d453067894f90a1b4645b1edd2eac"
format_stamp: "Formatted at 2026-06-20 04:25:14 on dist-test-slave-nk6d"
server_key: "f62282615da8757a6ff6a95b98b1e4ec"
server_key_iv: "9f077c56cec341a58085a5ba8caf1996"
server_key_version: "encryptionkey@0"
I20260620 04:25:14.998924 2797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929496330398-392-0/minicluster-data/master-0/wal/instance:
uuid: "f00d453067894f90a1b4645b1edd2eac"
format_stamp: "Formatted at 2026-06-20 04:25:14 on dist-test-slave-nk6d"
server_key: "f62282615da8757a6ff6a95b98b1e4ec"
server_key_iv: "9f077c56cec341a58085a5ba8caf1996"
server_key_version: "encryptionkey@0"
I20260620 04:25:15.002758 2797 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 04:25:15.005268 2812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:15.006561 2797 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 04:25:15.006672 2797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929496330398-392-0/minicluster-data/master-0/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929496330398-392-0/minicluster-data/master-0/wal
uuid: "f00d453067894f90a1b4645b1edd2eac"
format_stamp: "Formatted at 2026-06-20 04:25:14 on dist-test-slave-nk6d"
server_key: "f62282615da8757a6ff6a95b98b1e4ec"
server_key_iv: "9f077c56cec341a58085a5ba8caf1996"
server_key_version: "encryptionkey@0"
I20260620 04:25:15.006770 2797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929496330398-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929496330398-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929496330398-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:25:15.048465 2797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:25:15.051666 2797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:25:15.051868 2797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:25:15.058413 2797 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39795
I20260620 04:25:15.058422 2864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39795 every 8 connection(s)
I20260620 04:25:15.059367 2797 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:25:15.062028 2865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:25:15.062405 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 2797
I20260620 04:25:15.062496 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929496330398-392-0/minicluster-data/master-0/wal/instance
I20260620 04:25:15.062762 392 external_mini_cluster.cc:1428] Setting key dc08a84b77825f5045dc8371b29bcec6
I20260620 04:25:15.068545 2865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f00d453067894f90a1b4645b1edd2eac: Bootstrap starting.
I20260620 04:25:15.071099 2865 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f00d453067894f90a1b4645b1edd2eac: Neither blocks nor log segments found. Creating new log.
Jun 20 04:25:15 dist-test-slave-nk6d krb5kdc[2784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929514, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:25:15.071813 2865 log.cc:824] T 00000000000000000000000000000000 P f00d453067894f90a1b4645b1edd2eac: Log is configured to *not* fsync() on all Append() calls
I20260620 04:25:15.073619 2865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f00d453067894f90a1b4645b1edd2eac: No bootstrap required, opened a new log
I20260620 04:25:15.076143 2868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:15.064031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50334 (local address 127.0.98.62:39795)
0620 04:25:15.064428 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:15.064442 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:15.064476 (+ 34us) server_negotiation.cc:407] Connection header received
0620 04:25:15.065150 (+ 674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:15.065167 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:15.065487 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:15.065759 (+ 272us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:15.066588 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:15.067568 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:15.068692 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:15.069157 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:15.069280 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:15.071847 (+ 2567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:15.071852 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:15.071865 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:25:15.071891 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:15.073465 (+ 1574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:15.073876 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:15.073882 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:15.073886 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:15.073999 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:15.074245 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:15.074248 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:15.074250 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:15.074679 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:15.074860 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:15.075109 (+ 249us) server_negotiation.cc:299] Negotiation successful
0620 04:25:15.075382 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":108,"threads_started":1}
I20260620 04:25:15.077122 2865 raft_consensus.cc:348] T 00000000000000000000000000000000 P f00d453067894f90a1b4645b1edd2eac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f00d453067894f90a1b4645b1edd2eac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39795 } }
I20260620 04:25:15.077334 2865 raft_consensus.cc:374] T 00000000000000000000000000000000 P f00d453067894f90a1b4645b1edd2eac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:25:15.077394 2865 raft_consensus.cc:729] T 00000000000000000000000000000000 P f00d453067894f90a1b4645b1edd2eac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f00d453067894f90a1b4645b1edd2eac, State: Initialized, Role: FOLLOWER
I20260620 04:25:15.077817 2865 consensus_queue.cc:260] T 00000000000000000000000000000000 P f00d453067894f90a1b4645b1edd2eac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f00d453067894f90a1b4645b1edd2eac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39795 } }
I20260620 04:25:15.077945 2865 raft_consensus.cc:388] T 00000000000000000000000000000000 P f00d453067894f90a1b4645b1edd2eac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:25:15.078007 2865 raft_consensus.cc:482] T 00000000000000000000000000000000 P f00d453067894f90a1b4645b1edd2eac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:25:15.078101 2865 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f00d453067894f90a1b4645b1edd2eac [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:25:15.079000 2865 raft_consensus.cc:504] T 00000000000000000000000000000000 P f00d453067894f90a1b4645b1edd2eac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f00d453067894f90a1b4645b1edd2eac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39795 } }
I20260620 04:25:15.079316 2865 leader_election.cc:302] T 00000000000000000000000000000000 P f00d453067894f90a1b4645b1edd2eac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f00d453067894f90a1b4645b1edd2eac; no voters:
I20260620 04:25:15.079602 2865 leader_election.cc:288] T 00000000000000000000000000000000 P f00d453067894f90a1b4645b1edd2eac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:25:15.079722 2870 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f00d453067894f90a1b4645b1edd2eac [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:25:15.079942 2870 raft_consensus.cc:686] T 00000000000000000000000000000000 P f00d453067894f90a1b4645b1edd2eac [term 1 LEADER]: Becoming Leader. State: Replica: f00d453067894f90a1b4645b1edd2eac, State: Running, Role: LEADER
I20260620 04:25:15.080286 2870 consensus_queue.cc:237] T 00000000000000000000000000000000 P f00d453067894f90a1b4645b1edd2eac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f00d453067894f90a1b4645b1edd2eac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39795 } }
I20260620 04:25:15.080794 2865 sys_catalog.cc:564] T 00000000000000000000000000000000 P f00d453067894f90a1b4645b1edd2eac [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:25:15.081825 2872 sys_catalog.cc:455] T 00000000000000000000000000000000 P f00d453067894f90a1b4645b1edd2eac [sys.catalog]: SysCatalogTable state changed. Reason: New leader f00d453067894f90a1b4645b1edd2eac. Latest consensus state: current_term: 1 leader_uuid: "f00d453067894f90a1b4645b1edd2eac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f00d453067894f90a1b4645b1edd2eac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39795 } } }
I20260620 04:25:15.082177 2872 sys_catalog.cc:458] T 00000000000000000000000000000000 P f00d453067894f90a1b4645b1edd2eac [sys.catalog]: This master's current role is: LEADER
I20260620 04:25:15.082082 2871 sys_catalog.cc:455] T 00000000000000000000000000000000 P f00d453067894f90a1b4645b1edd2eac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f00d453067894f90a1b4645b1edd2eac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f00d453067894f90a1b4645b1edd2eac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39795 } } }
I20260620 04:25:15.082491 2871 sys_catalog.cc:458] T 00000000000000000000000000000000 P f00d453067894f90a1b4645b1edd2eac [sys.catalog]: This master's current role is: LEADER
I20260620 04:25:15.082587 2882 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:25:15.085323 2882 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:25:15.090281 2882 catalog_manager.cc:1347] Generated new cluster ID: f264eb75064547cbb14006c6f2cf1444
I20260620 04:25:15.090348 2882 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:25:15.104166 2882 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:25:15.105198 2882 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:25:15.111976 2882 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f00d453067894f90a1b4645b1edd2eac: Generated new TSK 0
I20260620 04:25:15.112685 2882 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929496330398-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929496330398-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 04:25:15.146162 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929496330398-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929496330398-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929496330398-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929496330398-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39795
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929496330398-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:25:15.253472 2891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:25:15.253715 2891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:25:15.253754 2891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:25:15.257637 2891 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:25:15.257686 2891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:25:15.257740 2891 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 04:25:15.262787 2891 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929496330398-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929496330398-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929496330398-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929496330398-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39795
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2891
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929496330398-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:25:15.263697 2891 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:25:15.264638 2891 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:25:15.273824 2899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:25:15.273844 2897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:25:15.273844 2896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:25:15.274185 2891 server_base.cc:1034] running on GCE node
I20260620 04:25:15.274648 2891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:25:15.275271 2891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:25:15.276414 2891 hybrid_clock.cc:648] HybridClock initialized: now 1781929515276395 us; error 35 us; skew 500 ppm
Jun 20 04:25:15 dist-test-slave-nk6d krb5kdc[2784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929515, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:25:15.278947 2891 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 04:25:15.279796 2891 webserver.cc:460] Webserver started at http://127.0.98.1:39867/ using document root <none> and password file <none>
I20260620 04:25:15.280323 2891 fs_manager.cc:359] Metadata directory not provided
I20260620 04:25:15.280378 2891 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:25:15.280529 2891 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:25:15.282509 2891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929496330398-392-0/minicluster-data/ts-0/data/instance:
uuid: "accff32f309840d8ad973594e7755cce"
format_stamp: "Formatted at 2026-06-20 04:25:15 on dist-test-slave-nk6d"
server_key: "d773db18ec759ae8cca1260c1ee8ec30"
server_key_iv: "f42728605ed5bdaa6a32768131b6cc19"
server_key_version: "encryptionkey@0"
I20260620 04:25:15.283000 2891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929496330398-392-0/minicluster-data/ts-0/wal/instance:
uuid: "accff32f309840d8ad973594e7755cce"
format_stamp: "Formatted at 2026-06-20 04:25:15 on dist-test-slave-nk6d"
server_key: "d773db18ec759ae8cca1260c1ee8ec30"
server_key_iv: "f42728605ed5bdaa6a32768131b6cc19"
server_key_version: "encryptionkey@0"
I20260620 04:25:15.286398 2891 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260620 04:25:15.288686 2906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:15.289860 2891 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:25:15.289965 2891 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929496330398-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929496330398-392-0/minicluster-data/ts-0/wal
uuid: "accff32f309840d8ad973594e7755cce"
format_stamp: "Formatted at 2026-06-20 04:25:15 on dist-test-slave-nk6d"
server_key: "d773db18ec759ae8cca1260c1ee8ec30"
server_key_iv: "f42728605ed5bdaa6a32768131b6cc19"
server_key_version: "encryptionkey@0"
I20260620 04:25:15.290053 2891 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929496330398-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929496330398-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929496330398-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:25:15.303905 2891 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:25:15.307055 2891 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:25:15.307235 2891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:25:15.307811 2891 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:25:15.308624 2891 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:25:15.308670 2891 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:15.308720 2891 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:25:15.308748 2891 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:15.317034 2891 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34743
I20260620 04:25:15.317071 3019 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34743 every 8 connection(s)
I20260620 04:25:15.317893 2891 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929496330398-392-0/minicluster-data/ts-0/data/info.pb
I20260620 04:25:15.321839 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 2891
I20260620 04:25:15.321933 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929496330398-392-0/minicluster-data/ts-0/wal/instance
I20260620 04:25:15.322181 392 external_mini_cluster.cc:1428] Setting key fd59f132c65fb0c2e68b0c2634c2c61a
Jun 20 04:25:15 dist-test-slave-nk6d krb5kdc[2784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929515, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 04:25:15.331203 2868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:15.319637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49931 (local address 127.0.98.62:39795)
0620 04:25:15.319762 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:15.319765 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:15.320604 (+ 839us) server_negotiation.cc:407] Connection header received
0620 04:25:15.321954 (+ 1350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:15.321957 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:15.322010 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:15.322086 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:15.323306 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:15.324055 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:15.325212 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:15.325443 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:15.325531 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:15.328022 (+ 2491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:15.328025 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:15.328027 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:25:15.328049 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:15.329327 (+ 1278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:15.329783 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:15.329787 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:15.329788 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:15.329859 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:15.330151 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:15.330154 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:15.330156 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:15.330320 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:15.330399 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:15.330878 (+ 479us) server_negotiation.cc:299] Negotiation successful
0620 04:25:15.331000 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 04:25:15.331991 3022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:15.319998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39795 (local address 127.0.98.1:49931)
0620 04:25:15.320453 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:15.320494 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:15.321756 (+ 1262us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:15.322250 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:15.322259 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:15.322688 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:15.323175 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:15.323188 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:15.324168 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:15.324171 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:15.325091 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:15.325099 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:15.325581 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:15.325585 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:15.325736 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:15.326307 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:15.326326 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:15.327848 (+ 1522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:25:15.329427 (+ 1579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:15.329433 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:15.329447 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:15.329696 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:15.329936 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:15.329938 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:15.329940 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:15.330069 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:15.330412 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:25:15.330418 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:25:15.330664 (+ 246us) client_negotiation.cc:769] Sending connection context
0620 04:25:15.330824 (+ 160us) client_negotiation.cc:240] Negotiation successful
0620 04:25:15.331129 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":108,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 04:25:15.333371 3020 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39795
I20260620 04:25:15.333652 3020 heartbeater.cc:464] Registering TS with master...
I20260620 04:25:15.334133 3020 heartbeater.cc:511] Master 127.0.98.62:39795 requested a full tablet report, sending...
I20260620 04:25:15.335675 2829 ts_manager.cc:194] Registered new tserver with Master: accff32f309840d8ad973594e7755cce (127.0.98.1:34743)
I20260620 04:25:15.337252 2829 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:49931
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929496330398-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929496330398-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 04:25:15.347056 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929496330398-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929496330398-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929496330398-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929496330398-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39795
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929496330398-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:25:15.455221 3025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:25:15.455494 3025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:25:15.455542 3025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:25:15.459829 3025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:25:15.459914 3025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:25:15.460031 3025 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 04:25:15.465760 3025 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929496330398-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929496330398-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929496330398-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929496330398-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39795
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3025
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929496330398-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:25:15.466796 3025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:25:15.467796 3025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:25:15.476806 3030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:25:15.476815 3033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:25:15.476854 3031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:25:15.476992 3025 server_base.cc:1034] running on GCE node
I20260620 04:25:15.477406 3025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:25:15.477978 3025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:25:15.479143 3025 hybrid_clock.cc:648] HybridClock initialized: now 1781929515479127 us; error 33 us; skew 500 ppm
Jun 20 04:25:15 dist-test-slave-nk6d krb5kdc[2784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929515, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:25:15.481762 3025 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 04:25:15.482630 3025 webserver.cc:460] Webserver started at http://127.0.98.2:46109/ using document root <none> and password file <none>
I20260620 04:25:15.483177 3025 fs_manager.cc:359] Metadata directory not provided
I20260620 04:25:15.483235 3025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:25:15.483392 3025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:25:15.485363 3025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929496330398-392-0/minicluster-data/ts-1/data/instance:
uuid: "cde678570c4a473a905cf32944eb9f38"
format_stamp: "Formatted at 2026-06-20 04:25:15 on dist-test-slave-nk6d"
server_key: "87ba8d9d9eb3c06955271ac600e00136"
server_key_iv: "b768043e387ea2bf4d758a3161d62aef"
server_key_version: "encryptionkey@0"
I20260620 04:25:15.485859 3025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929496330398-392-0/minicluster-data/ts-1/wal/instance:
uuid: "cde678570c4a473a905cf32944eb9f38"
format_stamp: "Formatted at 2026-06-20 04:25:15 on dist-test-slave-nk6d"
server_key: "87ba8d9d9eb3c06955271ac600e00136"
server_key_iv: "b768043e387ea2bf4d758a3161d62aef"
server_key_version: "encryptionkey@0"
I20260620 04:25:15.489348 3025 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 04:25:15.491691 3040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:15.492977 3025 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:25:15.493072 3025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929496330398-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929496330398-392-0/minicluster-data/ts-1/wal
uuid: "cde678570c4a473a905cf32944eb9f38"
format_stamp: "Formatted at 2026-06-20 04:25:15 on dist-test-slave-nk6d"
server_key: "87ba8d9d9eb3c06955271ac600e00136"
server_key_iv: "b768043e387ea2bf4d758a3161d62aef"
server_key_version: "encryptionkey@0"
I20260620 04:25:15.493208 3025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929496330398-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929496330398-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929496330398-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:25:15.502826 3025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:25:15.505791 3025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:25:15.505959 3025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:25:15.506523 3025 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:25:15.507297 3025 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:25:15.507340 3025 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:15.507382 3025 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:25:15.507411 3025 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:15.515800 3025 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42645
I20260620 04:25:15.515895 3153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42645 every 8 connection(s)
I20260620 04:25:15.516772 3025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929496330398-392-0/minicluster-data/ts-1/data/info.pb
I20260620 04:25:15.522702 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 3025
I20260620 04:25:15.522814 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929496330398-392-0/minicluster-data/ts-1/wal/instance
I20260620 04:25:15.523079 392 external_mini_cluster.cc:1428] Setting key ad90a7b7b499ea437f0d30ec2aca2b1c
Jun 20 04:25:15 dist-test-slave-nk6d krb5kdc[2784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929515, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:25:15.530274 2868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:15.518551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59721 (local address 127.0.98.62:39795)
0620 04:25:15.518710 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:15.518714 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:15.519496 (+ 782us) server_negotiation.cc:407] Connection header received
0620 04:25:15.520877 (+ 1381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:15.520880 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:15.520924 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:15.521029 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:15.522166 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:15.522864 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:15.524131 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:15.524374 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:15.524454 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:15.526959 (+ 2505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:15.526962 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:15.526964 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:25:15.526994 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:15.528291 (+ 1297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:15.528858 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:15.528860 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:15.528861 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:15.528928 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:15.529255 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:15.529257 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:15.529258 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:15.529414 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:15.529480 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:15.530062 (+ 582us) server_negotiation.cc:299] Negotiation successful
0620 04:25:15.530156 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 04:25:15.531390 3156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:15.518929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39795 (local address 127.0.98.2:59721)
0620 04:25:15.519351 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:15.519393 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:15.520672 (+ 1279us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:15.521185 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:15.521193 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:15.521544 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:15.522019 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:15.522032 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:15.522995 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:15.522997 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:15.524025 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:15.524034 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:15.524554 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:15.524559 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:15.524701 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:15.525266 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:15.525282 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:15.526781 (+ 1499us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:25:15.528398 (+ 1617us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:15.528403 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:15.528420 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:15.528768 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:15.529003 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:15.529006 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:15.529008 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:15.529177 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:15.529499 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:25:15.529505 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:25:15.529825 (+ 320us) client_negotiation.cc:769] Sending connection context
0620 04:25:15.530034 (+ 209us) client_negotiation.cc:240] Negotiation successful
0620 04:25:15.530359 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":107,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 04:25:15.532457 3154 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39795
I20260620 04:25:15.532682 3154 heartbeater.cc:464] Registering TS with master...
I20260620 04:25:15.533150 3154 heartbeater.cc:511] Master 127.0.98.62:39795 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260620 04:25:15.534202 2829 ts_manager.cc:194] Registered new tserver with Master: cde678570c4a473a905cf32944eb9f38 (127.0.98.2:42645)
I20260620 04:25:15.534747 2829 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:59721
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929496330398-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929496330398-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 04:25:15.547183 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929496330398-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929496330398-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929496330398-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929496330398-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39795
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929496330398-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:25:15.654739 3159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:25:15.654982 3159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:25:15.655025 3159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:25:15.659029 3159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:25:15.659088 3159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:25:15.659145 3159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 04:25:15.664376 3159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929496330398-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929496330398-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929496330398-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929496330398-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39795
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3159
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929496330398-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:25:15.665380 3159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:25:15.666380 3159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:25:15.675650 3165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:25:15.675704 3159 server_base.cc:1034] running on GCE node
W20260620 04:25:15.675659 3164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:25:15.675858 3167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:25:15.676287 3159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:25:15.676891 3159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:25:15.678081 3159 hybrid_clock.cc:648] HybridClock initialized: now 1781929515678065 us; error 26 us; skew 500 ppm
Jun 20 04:25:15 dist-test-slave-nk6d krb5kdc[2784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929515, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:25:15.680761 3159 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 04:25:15.681694 3159 webserver.cc:460] Webserver started at http://127.0.98.3:39961/ using document root <none> and password file <none>
I20260620 04:25:15.682268 3159 fs_manager.cc:359] Metadata directory not provided
I20260620 04:25:15.682328 3159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:25:15.682507 3159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:25:15.684540 3159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929496330398-392-0/minicluster-data/ts-2/data/instance:
uuid: "fadbb7d628f94a5a98ba82bd42cf579b"
format_stamp: "Formatted at 2026-06-20 04:25:15 on dist-test-slave-nk6d"
server_key: "aa3e6a64be55c9ff3e6fdf3da59e44e0"
server_key_iv: "a7609ac9066411c9e84678042efd65b6"
server_key_version: "encryptionkey@0"
I20260620 04:25:15.685083 3159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929496330398-392-0/minicluster-data/ts-2/wal/instance:
uuid: "fadbb7d628f94a5a98ba82bd42cf579b"
format_stamp: "Formatted at 2026-06-20 04:25:15 on dist-test-slave-nk6d"
server_key: "aa3e6a64be55c9ff3e6fdf3da59e44e0"
server_key_iv: "a7609ac9066411c9e84678042efd65b6"
server_key_version: "encryptionkey@0"
I20260620 04:25:15.688593 3159 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 04:25:15.690886 3174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:15.692111 3159 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:25:15.692211 3159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929496330398-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929496330398-392-0/minicluster-data/ts-2/wal
uuid: "fadbb7d628f94a5a98ba82bd42cf579b"
format_stamp: "Formatted at 2026-06-20 04:25:15 on dist-test-slave-nk6d"
server_key: "aa3e6a64be55c9ff3e6fdf3da59e44e0"
server_key_iv: "a7609ac9066411c9e84678042efd65b6"
server_key_version: "encryptionkey@0"
I20260620 04:25:15.692299 3159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929496330398-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929496330398-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929496330398-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:25:15.717429 3159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:25:15.720535 3159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:25:15.720705 3159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:25:15.721269 3159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:25:15.722065 3159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:25:15.722110 3159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:15.722151 3159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:25:15.722182 3159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:15.730082 3159 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39557
I20260620 04:25:15.730129 3287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39557 every 8 connection(s)
I20260620 04:25:15.730906 3159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929496330398-392-0/minicluster-data/ts-2/data/info.pb
I20260620 04:25:15.733170 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 3159
I20260620 04:25:15.733259 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929496330398-392-0/minicluster-data/ts-2/wal/instance
I20260620 04:25:15.733500 392 external_mini_cluster.cc:1428] Setting key 8014404e947fe3d51445f5178fb46eca
Jun 20 04:25:15 dist-test-slave-nk6d krb5kdc[2784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929515, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:25:15.743772 2868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:15.732660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47909 (local address 127.0.98.62:39795)
0620 04:25:15.732797 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:15.732801 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:15.733720 (+ 919us) server_negotiation.cc:407] Connection header received
0620 04:25:15.735187 (+ 1467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:15.735190 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:15.735242 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:15.735313 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:15.736481 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:15.736994 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:15.738091 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:15.738297 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:15.738380 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:15.740699 (+ 2319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:15.740702 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:15.740704 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:25:15.740727 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:15.741960 (+ 1233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:15.742409 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:15.742414 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:15.742415 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:15.742487 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:15.742781 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:15.742784 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:15.742786 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:15.742951 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:15.743031 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:15.743498 (+ 467us) server_negotiation.cc:299] Negotiation successful
0620 04:25:15.743615 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 04:25:15.744581 3290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:15.733087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39795 (local address 127.0.98.3:47909)
0620 04:25:15.733560 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:15.733610 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:15.734987 (+ 1377us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:15.735487 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:15.735494 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:15.735824 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:15.736357 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:15.736371 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:15.737103 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:15.737105 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:15.737979 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:15.737987 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:15.738421 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:15.738425 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:15.738581 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:15.739140 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:15.739156 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:15.740552 (+ 1396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:25:15.742065 (+ 1513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:15.742070 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:15.742086 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:15.742323 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:15.742569 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:15.742572 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:15.742574 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:15.742702 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:15.743047 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:25:15.743054 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:25:15.743288 (+ 234us) client_negotiation.cc:769] Sending connection context
0620 04:25:15.743450 (+ 162us) client_negotiation.cc:240] Negotiation successful
0620 04:25:15.743759 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":107,"threads_started":1}
I20260620 04:25:15.745645 3288 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39795
I20260620 04:25:15.745882 3288 heartbeater.cc:464] Registering TS with master...
I20260620 04:25:15.746405 3288 heartbeater.cc:511] Master 127.0.98.62:39795 requested a full tablet report, sending...
I20260620 04:25:15.747309 2829 ts_manager.cc:194] Registered new tserver with Master: fadbb7d628f94a5a98ba82bd42cf579b (127.0.98.3:39557)
I20260620 04:25:15.747833 2829 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47909
I20260620 04:25:15.758632 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 04:25:15.770720 2868 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:25:15.769393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50348 (local address 127.0.98.62:39795)
0620 04:25:15.769564 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:15.769568 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:15.769636 (+ 68us) server_negotiation.cc:407] Connection header received
0620 04:25:15.769758 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:15.769761 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:15.769806 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:15.769882 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:15.770577 (+ 695us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50348: BlockingRecv error: recv got EOF from 127.0.0.1:50348 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 04:25:15.771883 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 2891
I20260620 04:25:15.776404 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 3025
I20260620 04:25:15.780731 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 3159
I20260620 04:25:15.785377 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 2797
2026-06-20T04:25:15Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1990 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781929496330398-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:25:15 dist-test-slave-nk6d krb5kdc[3300](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:25:15 dist-test-slave-nk6d krb5kdc[3300](info): set up 2 sockets
Jun 20 04:25:15 dist-test-slave-nk6d krb5kdc[3300](info): commencing operation
krb5kdc: starting...
W20260620 04:25:16.727660 392 mini_kdc.cc:121] Time spent starting KDC: real 0.922s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781929496330398-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781929496330398-392-0/krb5kdc/test-user.keytab.
Jun 20 04:25:16 dist-test-slave-nk6d krb5kdc[3300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929516, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:25:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:25:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 04:25:16.838270 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781929496330398-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781929496330398-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781929496330398-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37587
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37587
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:25:16.945057 3313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:25:16.945348 3313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:25:16.945402 3313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:25:16.949407 3313 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:25:16.949463 3313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:25:16.949491 3313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:25:16.949510 3313 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:25:16.949541 3313 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:25:16.954759 3313 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781929496330398-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781929496330398-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37587
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37587
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3313
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781929496330398-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:25:16.955753 3313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:25:16.956750 3313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:25:16.964577 3321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:25:16.964733 3318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:25:16.964552 3319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:25:16.964748 3313 server_base.cc:1034] running on GCE node
I20260620 04:25:16.965440 3313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:25:16.966504 3313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:25:16.967736 3313 hybrid_clock.cc:648] HybridClock initialized: now 1781929516967715 us; error 36 us; skew 500 ppm
Jun 20 04:25:16 dist-test-slave-nk6d krb5kdc[3300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929516, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:25:16.970440 3313 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 04:25:16.971328 3313 webserver.cc:460] Webserver started at http://127.0.98.62:34657/ using document root <none> and password file <none>
I20260620 04:25:16.971839 3313 fs_manager.cc:359] Metadata directory not provided
I20260620 04:25:16.971901 3313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:25:16.972108 3313 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:25:16.974077 3313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781929496330398-392-0/minicluster-data/master-0/data/instance:
uuid: "841965f18b474939be8ca5ac21fe9230"
format_stamp: "Formatted at 2026-06-20 04:25:16 on dist-test-slave-nk6d"
server_key: "1e73347e2e3b7bb300cfccac9f579263"
server_key_iv: "e69aae8baca2a1356645f070cc045ec5"
server_key_version: "encryptionkey@0"
I20260620 04:25:16.974586 3313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781929496330398-392-0/minicluster-data/master-0/wal/instance:
uuid: "841965f18b474939be8ca5ac21fe9230"
format_stamp: "Formatted at 2026-06-20 04:25:16 on dist-test-slave-nk6d"
server_key: "1e73347e2e3b7bb300cfccac9f579263"
server_key_iv: "e69aae8baca2a1356645f070cc045ec5"
server_key_version: "encryptionkey@0"
I20260620 04:25:16.978176 3313 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 04:25:16.980506 3328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:16.981657 3313 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:25:16.981765 3313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781929496330398-392-0/minicluster-data/master-0/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781929496330398-392-0/minicluster-data/master-0/wal
uuid: "841965f18b474939be8ca5ac21fe9230"
format_stamp: "Formatted at 2026-06-20 04:25:16 on dist-test-slave-nk6d"
server_key: "1e73347e2e3b7bb300cfccac9f579263"
server_key_iv: "e69aae8baca2a1356645f070cc045ec5"
server_key_version: "encryptionkey@0"
I20260620 04:25:16.981858 3313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781929496330398-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781929496330398-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781929496330398-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:25:17.000100 3313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:25:17.003173 3313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:25:17.003353 3313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:25:17.010078 3313 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37587
I20260620 04:25:17.010344 3380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37587 every 8 connection(s)
I20260620 04:25:17.011001 3313 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:25:17.013666 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 3313
I20260620 04:25:17.013725 3381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:25:17.013764 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781929496330398-392-0/minicluster-data/master-0/wal/instance
I20260620 04:25:17.014029 392 external_mini_cluster.cc:1428] Setting key 34591e54041151992ae5e686b57db849
I20260620 04:25:17.020527 3381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 841965f18b474939be8ca5ac21fe9230: Bootstrap starting.
I20260620 04:25:17.022647 3381 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 841965f18b474939be8ca5ac21fe9230: Neither blocks nor log segments found. Creating new log.
Jun 20 04:25:17 dist-test-slave-nk6d krb5kdc[3300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929516, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:25:17.023402 3381 log.cc:824] T 00000000000000000000000000000000 P 841965f18b474939be8ca5ac21fe9230: Log is configured to *not* fsync() on all Append() calls
I20260620 04:25:17.025213 3381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 841965f18b474939be8ca5ac21fe9230: No bootstrap required, opened a new log
I20260620 04:25:17.027899 3384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:17.015344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45802 (local address 127.0.98.62:37587)
0620 04:25:17.015761 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:17.015782 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:17.015826 (+ 44us) server_negotiation.cc:407] Connection header received
0620 04:25:17.016866 (+ 1040us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:17.016891 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:17.017329 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:17.017711 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:17.018332 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:17.019795 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:17.020698 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:17.021188 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:17.021333 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:17.023708 (+ 2375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:17.023713 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:17.023727 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:25:17.023753 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:17.025317 (+ 1564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:17.025808 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:17.025812 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:17.025817 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:17.025922 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:17.026174 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:17.026176 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:17.026177 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:17.026526 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:17.026670 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:17.026937 (+ 267us) server_negotiation.cc:299] Negotiation successful
0620 04:25:17.027190 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":112,"threads_started":1}
I20260620 04:25:17.028970 3381 raft_consensus.cc:348] T 00000000000000000000000000000000 P 841965f18b474939be8ca5ac21fe9230 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "841965f18b474939be8ca5ac21fe9230" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37587 } }
I20260620 04:25:17.029232 3381 raft_consensus.cc:374] T 00000000000000000000000000000000 P 841965f18b474939be8ca5ac21fe9230 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:25:17.029285 3381 raft_consensus.cc:729] T 00000000000000000000000000000000 P 841965f18b474939be8ca5ac21fe9230 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 841965f18b474939be8ca5ac21fe9230, State: Initialized, Role: FOLLOWER
I20260620 04:25:17.029692 3381 consensus_queue.cc:260] T 00000000000000000000000000000000 P 841965f18b474939be8ca5ac21fe9230 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "841965f18b474939be8ca5ac21fe9230" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37587 } }
I20260620 04:25:17.029820 3381 raft_consensus.cc:388] T 00000000000000000000000000000000 P 841965f18b474939be8ca5ac21fe9230 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:25:17.029896 3381 raft_consensus.cc:482] T 00000000000000000000000000000000 P 841965f18b474939be8ca5ac21fe9230 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:25:17.029990 3381 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 841965f18b474939be8ca5ac21fe9230 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:25:17.030745 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 04:25:17.030970 3381 raft_consensus.cc:504] T 00000000000000000000000000000000 P 841965f18b474939be8ca5ac21fe9230 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "841965f18b474939be8ca5ac21fe9230" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37587 } }
I20260620 04:25:17.031275 3381 leader_election.cc:302] T 00000000000000000000000000000000 P 841965f18b474939be8ca5ac21fe9230 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 841965f18b474939be8ca5ac21fe9230; no voters:
I20260620 04:25:17.031544 3381 leader_election.cc:288] T 00000000000000000000000000000000 P 841965f18b474939be8ca5ac21fe9230 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:25:17.031621 3386 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 841965f18b474939be8ca5ac21fe9230 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:25:17.031777 3386 raft_consensus.cc:686] T 00000000000000000000000000000000 P 841965f18b474939be8ca5ac21fe9230 [term 1 LEADER]: Becoming Leader. State: Replica: 841965f18b474939be8ca5ac21fe9230, State: Running, Role: LEADER
I20260620 04:25:17.032132 3386 consensus_queue.cc:237] T 00000000000000000000000000000000 P 841965f18b474939be8ca5ac21fe9230 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "841965f18b474939be8ca5ac21fe9230" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37587 } }
I20260620 04:25:17.032761 3381 sys_catalog.cc:564] T 00000000000000000000000000000000 P 841965f18b474939be8ca5ac21fe9230 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:25:17.033711 3388 sys_catalog.cc:455] T 00000000000000000000000000000000 P 841965f18b474939be8ca5ac21fe9230 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 841965f18b474939be8ca5ac21fe9230. Latest consensus state: current_term: 1 leader_uuid: "841965f18b474939be8ca5ac21fe9230" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "841965f18b474939be8ca5ac21fe9230" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37587 } } }
I20260620 04:25:17.033833 3388 sys_catalog.cc:458] T 00000000000000000000000000000000 P 841965f18b474939be8ca5ac21fe9230 [sys.catalog]: This master's current role is: LEADER
I20260620 04:25:17.034114 3387 sys_catalog.cc:455] T 00000000000000000000000000000000 P 841965f18b474939be8ca5ac21fe9230 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "841965f18b474939be8ca5ac21fe9230" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "841965f18b474939be8ca5ac21fe9230" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37587 } } }
I20260620 04:25:17.034188 3387 sys_catalog.cc:458] T 00000000000000000000000000000000 P 841965f18b474939be8ca5ac21fe9230 [sys.catalog]: This master's current role is: LEADER
I20260620 04:25:17.034198 3396 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:25:17.037210 3396 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:25:17.042790 3396 catalog_manager.cc:1347] Generated new cluster ID: 466f9e46597e4b18a60cf4f07ddb8a5f
I20260620 04:25:17.042853 3396 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:25:17.052430 3384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:17.046836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45814 (local address 127.0.98.62:37587)
0620 04:25:17.046937 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:17.046941 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:17.047250 (+ 309us) server_negotiation.cc:407] Connection header received
0620 04:25:17.047360 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:17.047363 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:17.047415 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:17.047502 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:17.048087 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:17.048899 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:17.050023 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:17.050392 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:17.050485 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:17.051342 (+ 857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:17.051346 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:17.051349 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:25:17.051995 (+ 646us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:17.052033 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:25:17.052054 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:17.052139 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:17.052218 (+ 79us) server_negotiation.cc:299] Negotiation successful
0620 04:25:17.052280 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 04:25:17.067827 3396 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:25:17.069099 3396 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:25:17.082372 3396 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 841965f18b474939be8ca5ac21fe9230: Generated new TSK 0
I20260620 04:25:17.083254 3396 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 04:25:17.095939 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 3313
2026-06-20T04:25:17Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1309 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929496330398-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:25:17 dist-test-slave-nk6d krb5kdc[3414](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:25:17 dist-test-slave-nk6d krb5kdc[3414](info): set up 2 sockets
Jun 20 04:25:17 dist-test-slave-nk6d krb5kdc[3414](info): commencing operation
krb5kdc: starting...
W20260620 04:25:18.039646 392 mini_kdc.cc:121] Time spent starting KDC: real 0.925s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929496330398-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929496330398-392-0/krb5kdc/test-user.keytab.
Jun 20 04:25:18 dist-test-slave-nk6d krb5kdc[3414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929518, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:25:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:25:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 04:25:18.152410 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929496330398-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929496330398-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929496330398-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46485
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46485
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:25:18.259584 3427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:25:18.259847 3427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:25:18.259888 3427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:25:18.263859 3427 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:25:18.263918 3427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:25:18.263936 3427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:25:18.263976 3427 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:25:18.264007 3427 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:25:18.269233 3427 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929496330398-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929496330398-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46485
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46485
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3427
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929496330398-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:25:18.270218 3427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:25:18.271178 3427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:25:18.279322 3432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:25:18.279379 3435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:25:18.279322 3433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:25:18.279398 3427 server_base.cc:1034] running on GCE node
I20260620 04:25:18.280179 3427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:25:18.281361 3427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:25:18.282549 3427 hybrid_clock.cc:648] HybridClock initialized: now 1781929518282534 us; error 28 us; skew 500 ppm
Jun 20 04:25:18 dist-test-slave-nk6d krb5kdc[3414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929518, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:25:18.285173 3427 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 04:25:18.286108 3427 webserver.cc:460] Webserver started at http://127.0.98.62:40285/ using document root <none> and password file <none>
I20260620 04:25:18.286623 3427 fs_manager.cc:359] Metadata directory not provided
I20260620 04:25:18.286688 3427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:25:18.286886 3427 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:25:18.288944 3427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929496330398-392-0/minicluster-data/master-0/data/instance:
uuid: "91fde49b2da2456fbcb4e79d11042f08"
format_stamp: "Formatted at 2026-06-20 04:25:18 on dist-test-slave-nk6d"
server_key: "6ea26a3c9789f454c36958d2c1f2030a"
server_key_iv: "4ac5ec87c3ac1b45e9ffe2da51b673e6"
server_key_version: "encryptionkey@0"
I20260620 04:25:18.289453 3427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929496330398-392-0/minicluster-data/master-0/wal/instance:
uuid: "91fde49b2da2456fbcb4e79d11042f08"
format_stamp: "Formatted at 2026-06-20 04:25:18 on dist-test-slave-nk6d"
server_key: "6ea26a3c9789f454c36958d2c1f2030a"
server_key_iv: "4ac5ec87c3ac1b45e9ffe2da51b673e6"
server_key_version: "encryptionkey@0"
I20260620 04:25:18.293048 3427 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 04:25:18.295396 3442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:18.296748 3427 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:25:18.296921 3427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929496330398-392-0/minicluster-data/master-0/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929496330398-392-0/minicluster-data/master-0/wal
uuid: "91fde49b2da2456fbcb4e79d11042f08"
format_stamp: "Formatted at 2026-06-20 04:25:18 on dist-test-slave-nk6d"
server_key: "6ea26a3c9789f454c36958d2c1f2030a"
server_key_iv: "4ac5ec87c3ac1b45e9ffe2da51b673e6"
server_key_version: "encryptionkey@0"
I20260620 04:25:18.297042 3427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929496330398-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929496330398-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929496330398-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:25:18.333349 3427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:25:18.336390 3427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:25:18.336588 3427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:25:18.343740 3427 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46485
I20260620 04:25:18.344029 3494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46485 every 8 connection(s)
I20260620 04:25:18.344769 3427 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:25:18.347553 3495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:25:18.348328 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 3427
I20260620 04:25:18.348446 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929496330398-392-0/minicluster-data/master-0/wal/instance
I20260620 04:25:18.348718 392 external_mini_cluster.cc:1428] Setting key 44884016bda3de7ee94372f8ebd82920
I20260620 04:25:18.353955 3495 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91fde49b2da2456fbcb4e79d11042f08: Bootstrap starting.
I20260620 04:25:18.356506 3495 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 91fde49b2da2456fbcb4e79d11042f08: Neither blocks nor log segments found. Creating new log.
I20260620 04:25:18.357208 3495 log.cc:824] T 00000000000000000000000000000000 P 91fde49b2da2456fbcb4e79d11042f08: Log is configured to *not* fsync() on all Append() calls
Jun 20 04:25:18 dist-test-slave-nk6d krb5kdc[3414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929518, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:25:18.359428 3495 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91fde49b2da2456fbcb4e79d11042f08: No bootstrap required, opened a new log
I20260620 04:25:18.363840 3495 raft_consensus.cc:348] T 00000000000000000000000000000000 P 91fde49b2da2456fbcb4e79d11042f08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91fde49b2da2456fbcb4e79d11042f08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46485 } }
I20260620 04:25:18.364136 3495 raft_consensus.cc:374] T 00000000000000000000000000000000 P 91fde49b2da2456fbcb4e79d11042f08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:25:18.364189 3495 raft_consensus.cc:729] T 00000000000000000000000000000000 P 91fde49b2da2456fbcb4e79d11042f08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91fde49b2da2456fbcb4e79d11042f08, State: Initialized, Role: FOLLOWER
I20260620 04:25:18.364329 3498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:18.350144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54728 (local address 127.0.98.62:46485)
0620 04:25:18.350592 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:18.350611 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:18.350659 (+ 48us) server_negotiation.cc:407] Connection header received
0620 04:25:18.351439 (+ 780us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:18.351474 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:18.351886 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:18.352279 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:18.353010 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:18.354411 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:18.355406 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:18.356122 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:18.356271 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:18.359066 (+ 2795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:18.359072 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:18.359085 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:25:18.359114 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:18.361069 (+ 1955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:18.361539 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:18.361545 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:18.361552 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:18.361705 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:18.362042 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:18.362045 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:18.362047 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:18.362540 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:18.362725 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:18.363018 (+ 293us) server_negotiation.cc:299] Negotiation successful
0620 04:25:18.363322 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":121,"threads_started":1}
I20260620 04:25:18.364679 3495 consensus_queue.cc:260] T 00000000000000000000000000000000 P 91fde49b2da2456fbcb4e79d11042f08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91fde49b2da2456fbcb4e79d11042f08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46485 } }
I20260620 04:25:18.364820 3495 raft_consensus.cc:388] T 00000000000000000000000000000000 P 91fde49b2da2456fbcb4e79d11042f08 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:25:18.364882 3495 raft_consensus.cc:482] T 00000000000000000000000000000000 P 91fde49b2da2456fbcb4e79d11042f08 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:25:18.364984 3495 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 91fde49b2da2456fbcb4e79d11042f08 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:25:18.365881 3495 raft_consensus.cc:504] T 00000000000000000000000000000000 P 91fde49b2da2456fbcb4e79d11042f08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91fde49b2da2456fbcb4e79d11042f08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46485 } }
I20260620 04:25:18.366210 3495 leader_election.cc:302] T 00000000000000000000000000000000 P 91fde49b2da2456fbcb4e79d11042f08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 91fde49b2da2456fbcb4e79d11042f08; no voters:
I20260620 04:25:18.366519 3495 leader_election.cc:288] T 00000000000000000000000000000000 P 91fde49b2da2456fbcb4e79d11042f08 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:25:18.366606 3500 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 91fde49b2da2456fbcb4e79d11042f08 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:25:18.366783 3500 raft_consensus.cc:686] T 00000000000000000000000000000000 P 91fde49b2da2456fbcb4e79d11042f08 [term 1 LEADER]: Becoming Leader. State: Replica: 91fde49b2da2456fbcb4e79d11042f08, State: Running, Role: LEADER
I20260620 04:25:18.367081 3500 consensus_queue.cc:237] T 00000000000000000000000000000000 P 91fde49b2da2456fbcb4e79d11042f08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91fde49b2da2456fbcb4e79d11042f08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46485 } }
I20260620 04:25:18.367803 3495 sys_catalog.cc:564] T 00000000000000000000000000000000 P 91fde49b2da2456fbcb4e79d11042f08 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:25:18.368769 3501 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91fde49b2da2456fbcb4e79d11042f08 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "91fde49b2da2456fbcb4e79d11042f08" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91fde49b2da2456fbcb4e79d11042f08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46485 } } }
I20260620 04:25:18.368906 3501 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91fde49b2da2456fbcb4e79d11042f08 [sys.catalog]: This master's current role is: LEADER
I20260620 04:25:18.369228 3502 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91fde49b2da2456fbcb4e79d11042f08 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 91fde49b2da2456fbcb4e79d11042f08. Latest consensus state: current_term: 1 leader_uuid: "91fde49b2da2456fbcb4e79d11042f08" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91fde49b2da2456fbcb4e79d11042f08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46485 } } }
I20260620 04:25:18.369307 3509 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:25:18.372617 3509 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:25:18.374387 3502 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91fde49b2da2456fbcb4e79d11042f08 [sys.catalog]: This master's current role is: LEADER
I20260620 04:25:18.377864 3509 catalog_manager.cc:1347] Generated new cluster ID: e54cd123570d48e29fd394b8a585a076
I20260620 04:25:18.377972 3509 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:25:18.390149 3509 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:25:18.391050 3509 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:25:18.397475 3509 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 91fde49b2da2456fbcb4e79d11042f08: Generated new TSK 0
I20260620 04:25:18.398150 3509 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929496330398-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929496330398-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 04:25:18.427013 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929496330398-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929496330398-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929496330398-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929496330398-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46485
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929496330398-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:25:18.536881 3521 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:25:18.537150 3521 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:25:18.537191 3521 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:25:18.541157 3521 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:25:18.541213 3521 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:25:18.541271 3521 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 04:25:18.546438 3521 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929496330398-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929496330398-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929496330398-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929496330398-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46485
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3521
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929496330398-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:25:18.547422 3521 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:25:18.548408 3521 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:25:18.557806 3529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:25:18.557821 3527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:25:18.557821 3526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:25:18.558229 3521 server_base.cc:1034] running on GCE node
I20260620 04:25:18.558597 3521 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:25:18.559194 3521 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:25:18.560338 3521 hybrid_clock.cc:648] HybridClock initialized: now 1781929518560305 us; error 48 us; skew 500 ppm
Jun 20 04:25:18 dist-test-slave-nk6d krb5kdc[3414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929518, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:25:18.562886 3521 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 04:25:18.563673 3521 webserver.cc:460] Webserver started at http://127.0.98.1:44245/ using document root <none> and password file <none>
I20260620 04:25:18.564209 3521 fs_manager.cc:359] Metadata directory not provided
I20260620 04:25:18.564266 3521 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:25:18.564422 3521 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:25:18.566358 3521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929496330398-392-0/minicluster-data/ts-0/data/instance:
uuid: "8f6d11c8e97246ab912e79984d5f70f6"
format_stamp: "Formatted at 2026-06-20 04:25:18 on dist-test-slave-nk6d"
server_key: "364356595894460291eab2fbc50c1b75"
server_key_iv: "aa2c4ece7bacc834bb5d93da0c1ed00d"
server_key_version: "encryptionkey@0"
I20260620 04:25:18.566838 3521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929496330398-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8f6d11c8e97246ab912e79984d5f70f6"
format_stamp: "Formatted at 2026-06-20 04:25:18 on dist-test-slave-nk6d"
server_key: "364356595894460291eab2fbc50c1b75"
server_key_iv: "aa2c4ece7bacc834bb5d93da0c1ed00d"
server_key_version: "encryptionkey@0"
I20260620 04:25:18.570515 3521 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 04:25:18.572839 3536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:18.574080 3521 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:25:18.574214 3521 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929496330398-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929496330398-392-0/minicluster-data/ts-0/wal
uuid: "8f6d11c8e97246ab912e79984d5f70f6"
format_stamp: "Formatted at 2026-06-20 04:25:18 on dist-test-slave-nk6d"
server_key: "364356595894460291eab2fbc50c1b75"
server_key_iv: "aa2c4ece7bacc834bb5d93da0c1ed00d"
server_key_version: "encryptionkey@0"
I20260620 04:25:18.574303 3521 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929496330398-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929496330398-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929496330398-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:25:18.592325 3521 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:25:18.595259 3521 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:25:18.595435 3521 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:25:18.596020 3521 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:25:18.596807 3521 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:25:18.596853 3521 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:18.596903 3521 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:25:18.596953 3521 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:18.605388 3521 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45861
I20260620 04:25:18.605434 3649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45861 every 8 connection(s)
I20260620 04:25:18.606279 3521 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929496330398-392-0/minicluster-data/ts-0/data/info.pb
I20260620 04:25:18.612747 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 3521
I20260620 04:25:18.612843 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929496330398-392-0/minicluster-data/ts-0/wal/instance
I20260620 04:25:18.613091 392 external_mini_cluster.cc:1428] Setting key 1c697c7372be6c28bbc098d1ef26315f
Jun 20 04:25:18 dist-test-slave-nk6d krb5kdc[3414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929518, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:25:18.619913 3498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:18.608091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53803 (local address 127.0.98.62:46485)
0620 04:25:18.608210 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:18.608214 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:18.609074 (+ 860us) server_negotiation.cc:407] Connection header received
0620 04:25:18.610368 (+ 1294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:18.610371 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:18.610418 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:18.610514 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:18.611644 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:18.612245 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:18.613464 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:18.613745 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:18.613827 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:18.616523 (+ 2696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:18.616526 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:18.616528 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:25:18.616551 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:18.617962 (+ 1411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:18.618429 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:18.618433 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:18.618435 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:18.618528 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:18.618831 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:18.618834 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:18.618836 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:18.619056 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:18.619154 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:18.619616 (+ 462us) server_negotiation.cc:299] Negotiation successful
0620 04:25:18.619745 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 04:25:18.620672 3652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:18.608485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46485 (local address 127.0.98.1:53803)
0620 04:25:18.608922 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:18.608964 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:18.610179 (+ 1215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:18.610658 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:18.610664 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:18.611012 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:18.611511 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:18.611525 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:18.612354 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:18.612357 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:18.613321 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:18.613330 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:18.613886 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:18.613890 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:18.614109 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:18.614696 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:18.614715 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:18.616352 (+ 1637us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:25:18.618083 (+ 1731us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:18.618088 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:18.618100 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:18.618338 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:18.618615 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:18.618617 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:18.618618 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:18.618749 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:18.619140 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:25:18.619145 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:25:18.619368 (+ 223us) client_negotiation.cc:769] Sending connection context
0620 04:25:18.619538 (+ 170us) client_negotiation.cc:240] Negotiation successful
0620 04:25:18.619791 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":115,"threads_started":1}
I20260620 04:25:18.621824 3650 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46485
I20260620 04:25:18.622092 3650 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 04:25:18.622660 3650 heartbeater.cc:511] Master 127.0.98.62:46485 requested a full tablet report, sending...
I20260620 04:25:18.624195 3459 ts_manager.cc:194] Registered new tserver with Master: 8f6d11c8e97246ab912e79984d5f70f6 (127.0.98.1:45861)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 04:25:18.625427 3459 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:53803
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929496330398-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929496330398-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 04:25:18.639509 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929496330398-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929496330398-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929496330398-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929496330398-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46485
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929496330398-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:25:18.745610 3655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:25:18.745862 3655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:25:18.745922 3655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:25:18.749878 3655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:25:18.749935 3655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:25:18.750012 3655 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 04:25:18.755203 3655 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929496330398-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929496330398-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929496330398-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929496330398-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46485
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3655
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929496330398-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:25:18.756220 3655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:25:18.757203 3655 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:25:18.766275 3660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:25:18.766302 3663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:25:18.766523 3655 server_base.cc:1034] running on GCE node
W20260620 04:25:18.766332 3661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:25:18.766959 3655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:25:18.767609 3655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:25:18.768771 3655 hybrid_clock.cc:648] HybridClock initialized: now 1781929518768749 us; error 30 us; skew 500 ppm
Jun 20 04:25:18 dist-test-slave-nk6d krb5kdc[3414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929518, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:25:18.771553 3655 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 04:25:18.772420 3655 webserver.cc:460] Webserver started at http://127.0.98.2:39227/ using document root <none> and password file <none>
I20260620 04:25:18.772934 3655 fs_manager.cc:359] Metadata directory not provided
I20260620 04:25:18.772994 3655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:25:18.773214 3655 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:25:18.775169 3655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929496330398-392-0/minicluster-data/ts-1/data/instance:
uuid: "a8e992740ff545cdb6d28dd587ae3726"
format_stamp: "Formatted at 2026-06-20 04:25:18 on dist-test-slave-nk6d"
server_key: "9278d40cd243e620cdf9fa944fa7f576"
server_key_iv: "17b6dc0f18feec0bace5ed2cb2c17268"
server_key_version: "encryptionkey@0"
I20260620 04:25:18.775691 3655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929496330398-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a8e992740ff545cdb6d28dd587ae3726"
format_stamp: "Formatted at 2026-06-20 04:25:18 on dist-test-slave-nk6d"
server_key: "9278d40cd243e620cdf9fa944fa7f576"
server_key_iv: "17b6dc0f18feec0bace5ed2cb2c17268"
server_key_version: "encryptionkey@0"
I20260620 04:25:18.779328 3655 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 04:25:18.781767 3670 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:18.783046 3655 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260620 04:25:18.783147 3655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929496330398-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929496330398-392-0/minicluster-data/ts-1/wal
uuid: "a8e992740ff545cdb6d28dd587ae3726"
format_stamp: "Formatted at 2026-06-20 04:25:18 on dist-test-slave-nk6d"
server_key: "9278d40cd243e620cdf9fa944fa7f576"
server_key_iv: "17b6dc0f18feec0bace5ed2cb2c17268"
server_key_version: "encryptionkey@0"
I20260620 04:25:18.783237 3655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929496330398-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929496330398-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929496330398-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:25:18.794417 3655 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:25:18.797276 3655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:25:18.797432 3655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:25:18.797958 3655 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:25:18.798730 3655 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:25:18.798774 3655 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:18.798825 3655 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:25:18.798849 3655 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:18.807063 3655 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39989
I20260620 04:25:18.807092 3783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39989 every 8 connection(s)
I20260620 04:25:18.807919 3655 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929496330398-392-0/minicluster-data/ts-1/data/info.pb
I20260620 04:25:18.815541 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 3655
I20260620 04:25:18.815668 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929496330398-392-0/minicluster-data/ts-1/wal/instance
I20260620 04:25:18.816063 392 external_mini_cluster.cc:1428] Setting key b852fe26f869cc0ae7d3d0be658ddf5c
Jun 20 04:25:18 dist-test-slave-nk6d krb5kdc[3414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929518, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:25:18.823217 3498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:18.809644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45793 (local address 127.0.98.62:46485)
0620 04:25:18.809770 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:18.809773 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:18.810582 (+ 809us) server_negotiation.cc:407] Connection header received
0620 04:25:18.811917 (+ 1335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:18.811921 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:18.812030 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:18.812101 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:18.813250 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:18.813756 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:18.814863 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:18.815107 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:18.815206 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:18.819546 (+ 4340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:18.819549 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:18.819551 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:25:18.819575 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:18.820884 (+ 1309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:18.821477 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:18.821480 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:18.821481 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:18.821554 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:18.821983 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:18.821986 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:18.821987 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:18.822193 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:18.822284 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:18.822992 (+ 708us) server_negotiation.cc:299] Negotiation successful
0620 04:25:18.823098 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 04:25:18.824414 3786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:18.810020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46485 (local address 127.0.98.2:45793)
0620 04:25:18.810437 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:18.810478 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:18.811712 (+ 1234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:18.812286 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:18.812294 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:18.812647 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:18.813104 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:18.813127 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:18.813868 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:18.813871 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:18.814748 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:18.814756 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:18.815844 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:18.815852 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:18.816102 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:18.816920 (+ 818us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:18.816941 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:18.819367 (+ 2426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:25:18.821005 (+ 1638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:18.821012 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:18.821031 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:18.821384 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:18.821634 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:18.821637 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:18.821639 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:18.821824 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:18.822301 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:25:18.822309 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:25:18.822676 (+ 367us) client_negotiation.cc:769] Sending connection context
0620 04:25:18.822913 (+ 237us) client_negotiation.cc:240] Negotiation successful
0620 04:25:18.823298 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":103,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 04:25:18.825656 3784 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46485
I20260620 04:25:18.825881 3784 heartbeater.cc:464] Registering TS with master...
I20260620 04:25:18.826393 3784 heartbeater.cc:511] Master 127.0.98.62:46485 requested a full tablet report, sending...
I20260620 04:25:18.827415 3459 ts_manager.cc:194] Registered new tserver with Master: a8e992740ff545cdb6d28dd587ae3726 (127.0.98.2:39989)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260620 04:25:18.828141 3459 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:45793
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929496330398-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929496330398-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 04:25:18.841012 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929496330398-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929496330398-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929496330398-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929496330398-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46485
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929496330398-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:25:18.947728 3789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:25:18.947992 3789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:25:18.948033 3789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:25:18.951860 3789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:25:18.951907 3789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:25:18.951982 3789 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 04:25:18.956947 3789 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929496330398-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929496330398-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929496330398-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929496330398-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46485
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3789
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929496330398-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:25:18.957832 3789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:25:18.958724 3789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:25:18.967644 3797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:25:18.967650 3794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:25:18.967694 3795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:25:18.967844 3789 server_base.cc:1034] running on GCE node
I20260620 04:25:18.968338 3789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:25:18.968919 3789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:25:18.970072 3789 hybrid_clock.cc:648] HybridClock initialized: now 1781929518970047 us; error 35 us; skew 500 ppm
Jun 20 04:25:18 dist-test-slave-nk6d krb5kdc[3414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929518, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:25:18.972544 3789 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 04:25:18.973385 3789 webserver.cc:460] Webserver started at http://127.0.98.3:35329/ using document root <none> and password file <none>
I20260620 04:25:18.974047 3789 fs_manager.cc:359] Metadata directory not provided
I20260620 04:25:18.974128 3789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:25:18.974350 3789 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:25:18.976521 3789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929496330398-392-0/minicluster-data/ts-2/data/instance:
uuid: "7733823b558143a291bae2cb438c397e"
format_stamp: "Formatted at 2026-06-20 04:25:18 on dist-test-slave-nk6d"
server_key: "8cdcf35d96a7aaef4105fc49e85f17c5"
server_key_iv: "21a7d8c60611e29ca5279a6ebda152d7"
server_key_version: "encryptionkey@0"
I20260620 04:25:18.977011 3789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929496330398-392-0/minicluster-data/ts-2/wal/instance:
uuid: "7733823b558143a291bae2cb438c397e"
format_stamp: "Formatted at 2026-06-20 04:25:18 on dist-test-slave-nk6d"
server_key: "8cdcf35d96a7aaef4105fc49e85f17c5"
server_key_iv: "21a7d8c60611e29ca5279a6ebda152d7"
server_key_version: "encryptionkey@0"
I20260620 04:25:18.980597 3789 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:25:18.983064 3804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:18.984272 3789 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:25:18.984375 3789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929496330398-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929496330398-392-0/minicluster-data/ts-2/wal
uuid: "7733823b558143a291bae2cb438c397e"
format_stamp: "Formatted at 2026-06-20 04:25:18 on dist-test-slave-nk6d"
server_key: "8cdcf35d96a7aaef4105fc49e85f17c5"
server_key_iv: "21a7d8c60611e29ca5279a6ebda152d7"
server_key_version: "encryptionkey@0"
I20260620 04:25:18.984534 3789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929496330398-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929496330398-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929496330398-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:25:19.001995 3789 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:25:19.005180 3789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:25:19.005375 3789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:25:19.005991 3789 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:25:19.006814 3789 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:25:19.006866 3789 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:19.006920 3789 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:25:19.006945 3789 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:19.015377 3789 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43773
I20260620 04:25:19.015443 3917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43773 every 8 connection(s)
I20260620 04:25:19.016294 3789 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929496330398-392-0/minicluster-data/ts-2/data/info.pb
I20260620 04:25:19.016660 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 3789
I20260620 04:25:19.016765 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929496330398-392-0/minicluster-data/ts-2/wal/instance
I20260620 04:25:19.017086 392 external_mini_cluster.cc:1428] Setting key a6f6d977bc8d80c56b2fd663c2753def
Jun 20 04:25:19 dist-test-slave-nk6d krb5kdc[3414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929518, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:25:19.029364 3498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:19.018094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42457 (local address 127.0.98.62:46485)
0620 04:25:19.018209 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:19.018213 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:19.019230 (+ 1017us) server_negotiation.cc:407] Connection header received
0620 04:25:19.020632 (+ 1402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:19.020635 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:19.020687 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:19.020758 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:19.021932 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:19.022453 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:19.023541 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:19.023750 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:19.023833 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:19.026218 (+ 2385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:19.026222 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:19.026223 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:25:19.026248 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:19.027569 (+ 1321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:19.028082 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:19.028085 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:19.028086 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:19.028157 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:19.028415 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:19.028417 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:19.028418 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:19.028574 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:19.028641 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:19.029147 (+ 506us) server_negotiation.cc:299] Negotiation successful
0620 04:25:19.029244 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 04:25:19.030439 3920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:19.018612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46485 (local address 127.0.98.3:42457)
0620 04:25:19.019082 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:19.019138 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:19.020431 (+ 1293us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:19.020940 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:19.020947 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:19.021304 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:19.021799 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:19.021813 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:19.022572 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:19.022575 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:19.023430 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:19.023439 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:19.023879 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:19.023883 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:19.024069 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:19.024624 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:19.024640 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:19.026047 (+ 1407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:25:19.027667 (+ 1620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:19.027672 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:19.027683 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:19.027940 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:19.028223 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:19.028225 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:19.028227 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:19.028349 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:19.028683 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:25:19.028688 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:25:19.028944 (+ 256us) client_negotiation.cc:769] Sending connection context
0620 04:25:19.029110 (+ 166us) client_negotiation.cc:240] Negotiation successful
0620 04:25:19.029445 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":106,"threads_started":1}
I20260620 04:25:19.031586 3918 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46485
I20260620 04:25:19.031811 3918 heartbeater.cc:464] Registering TS with master...
I20260620 04:25:19.032351 3918 heartbeater.cc:511] Master 127.0.98.62:46485 requested a full tablet report, sending...
I20260620 04:25:19.033267 3459 ts_manager.cc:194] Registered new tserver with Master: 7733823b558143a291bae2cb438c397e (127.0.98.3:43773)
I20260620 04:25:19.033815 3459 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:42457
I20260620 04:25:19.042644 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 04:25:19 dist-test-slave-nk6d krb5kdc[3414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929519, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 20 04:25:19 dist-test-slave-nk6d krb5kdc[3414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929519, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:25:19.066102 3498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:19.058168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54740 (local address 127.0.98.62:46485)
0620 04:25:19.058338 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:19.058343 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:19.058438 (+ 95us) server_negotiation.cc:407] Connection header received
0620 04:25:19.058581 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:19.058585 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:19.058631 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:19.058710 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:19.059395 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:19.059895 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:19.060806 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:19.061036 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:19.061128 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:19.063407 (+ 2279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:19.063410 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:19.063412 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:25:19.063436 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:19.064688 (+ 1252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:19.065075 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:19.065077 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:19.065079 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:19.065149 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:19.065339 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:19.065341 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:19.065342 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:19.065556 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:19.065668 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:19.065897 (+ 229us) server_negotiation.cc:299] Negotiation successful
0620 04:25:19.065985 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260620 04:25:19.066485 3459 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:54740
I20260620 04:25:19.067253 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 3521
I20260620 04:25:19.072751 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 3655
I20260620 04:25:19.077090 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 3789
I20260620 04:25:19.081750 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 3427
2026-06-20T04:25:19Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1987 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929496330398-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:25:19 dist-test-slave-nk6d krb5kdc[3930](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:25:19 dist-test-slave-nk6d krb5kdc[3930](info): set up 2 sockets
Jun 20 04:25:19 dist-test-slave-nk6d krb5kdc[3930](info): commencing operation
krb5kdc: starting...
W20260620 04:25:20.027837 392 mini_kdc.cc:121] Time spent starting KDC: real 0.926s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929496330398-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929496330398-392-0/krb5kdc/test-user.keytab.
Jun 20 04:25:20 dist-test-slave-nk6d krb5kdc[3930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929520, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:25:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:25:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 04:25:20.138675 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929496330398-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929496330398-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929496330398-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46285
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46285
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:25:20.243651 3943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:25:20.243916 3943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:25:20.243992 3943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:25:20.248332 3943 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:25:20.248386 3943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:25:20.248412 3943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:25:20.248433 3943 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:25:20.248454 3943 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:25:20.253669 3943 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929496330398-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929496330398-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46285
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46285
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3943
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929496330398-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:25:20.254660 3943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:25:20.255635 3943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:25:20.263759 3949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:25:20.263767 3951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:25:20.263768 3948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:25:20.264068 3943 server_base.cc:1034] running on GCE node
I20260620 04:25:20.264683 3943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:25:20.265758 3943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:25:20.266923 3943 hybrid_clock.cc:648] HybridClock initialized: now 1781929520266901 us; error 32 us; skew 500 ppm
Jun 20 04:25:20 dist-test-slave-nk6d krb5kdc[3930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929520, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:25:20.272958 3943 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 04:25:20.273887 3943 webserver.cc:460] Webserver started at http://127.0.98.62:34139/ using document root <none> and password file <none>
I20260620 04:25:20.274426 3943 fs_manager.cc:359] Metadata directory not provided
I20260620 04:25:20.274489 3943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:25:20.274672 3943 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:25:20.276710 3943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929496330398-392-0/minicluster-data/master-0/data/instance:
uuid: "a30949655f674e35942f127ba697f631"
format_stamp: "Formatted at 2026-06-20 04:25:20 on dist-test-slave-nk6d"
server_key: "228dd05c946d59def200a7c75ab45c16"
server_key_iv: "7c2fcc9bdaf8dfbcce0c692dc9582c8c"
server_key_version: "encryptionkey@0"
I20260620 04:25:20.277215 3943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929496330398-392-0/minicluster-data/master-0/wal/instance:
uuid: "a30949655f674e35942f127ba697f631"
format_stamp: "Formatted at 2026-06-20 04:25:20 on dist-test-slave-nk6d"
server_key: "228dd05c946d59def200a7c75ab45c16"
server_key_iv: "7c2fcc9bdaf8dfbcce0c692dc9582c8c"
server_key_version: "encryptionkey@0"
I20260620 04:25:20.280664 3943 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 04:25:20.282888 3958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:20.284062 3943 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:25:20.284174 3943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929496330398-392-0/minicluster-data/master-0/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929496330398-392-0/minicluster-data/master-0/wal
uuid: "a30949655f674e35942f127ba697f631"
format_stamp: "Formatted at 2026-06-20 04:25:20 on dist-test-slave-nk6d"
server_key: "228dd05c946d59def200a7c75ab45c16"
server_key_iv: "7c2fcc9bdaf8dfbcce0c692dc9582c8c"
server_key_version: "encryptionkey@0"
I20260620 04:25:20.284260 3943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929496330398-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929496330398-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929496330398-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:25:20.322218 3943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:25:20.325140 3943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:25:20.325315 3943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:25:20.331907 3943 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46285
I20260620 04:25:20.331923 4010 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46285 every 8 connection(s)
I20260620 04:25:20.332824 3943 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:25:20.334445 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 3943
I20260620 04:25:20.334559 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929496330398-392-0/minicluster-data/master-0/wal/instance
I20260620 04:25:20.334873 392 external_mini_cluster.cc:1428] Setting key 08a7fa76be4773f4d82a8ded709e763c
I20260620 04:25:20.335772 4011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:25:20.341174 4011 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a30949655f674e35942f127ba697f631: Bootstrap starting.
Jun 20 04:25:20 dist-test-slave-nk6d krb5kdc[3930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929520, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:25:20.343528 4011 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a30949655f674e35942f127ba697f631: Neither blocks nor log segments found. Creating new log.
I20260620 04:25:20.344290 4011 log.cc:824] T 00000000000000000000000000000000 P a30949655f674e35942f127ba697f631: Log is configured to *not* fsync() on all Append() calls
I20260620 04:25:20.346294 4011 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a30949655f674e35942f127ba697f631: No bootstrap required, opened a new log
I20260620 04:25:20.348214 4014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:20.336267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40938 (local address 127.0.98.62:46285)
0620 04:25:20.336701 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:20.336716 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:20.336754 (+ 38us) server_negotiation.cc:407] Connection header received
0620 04:25:20.337387 (+ 633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:20.337404 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:20.337736 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:20.338046 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:20.338815 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:20.339791 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:20.340674 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:20.341140 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:20.341280 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:20.343763 (+ 2483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:20.343768 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:20.343781 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:25:20.343808 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:20.345500 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:20.345982 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:20.345986 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:20.345990 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:20.346089 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:20.346343 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:20.346346 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:20.346347 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:20.346747 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:20.346883 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:20.347174 (+ 291us) server_negotiation.cc:299] Negotiation successful
0620 04:25:20.347439 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":130,"threads_started":1}
I20260620 04:25:20.350003 4011 raft_consensus.cc:348] T 00000000000000000000000000000000 P a30949655f674e35942f127ba697f631 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a30949655f674e35942f127ba697f631" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46285 } }
I20260620 04:25:20.350251 4011 raft_consensus.cc:374] T 00000000000000000000000000000000 P a30949655f674e35942f127ba697f631 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:25:20.350332 4011 raft_consensus.cc:729] T 00000000000000000000000000000000 P a30949655f674e35942f127ba697f631 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a30949655f674e35942f127ba697f631, State: Initialized, Role: FOLLOWER
I20260620 04:25:20.350737 4011 consensus_queue.cc:260] T 00000000000000000000000000000000 P a30949655f674e35942f127ba697f631 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a30949655f674e35942f127ba697f631" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46285 } }
I20260620 04:25:20.350860 4011 raft_consensus.cc:388] T 00000000000000000000000000000000 P a30949655f674e35942f127ba697f631 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:25:20.350921 4011 raft_consensus.cc:482] T 00000000000000000000000000000000 P a30949655f674e35942f127ba697f631 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:25:20.351012 4011 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a30949655f674e35942f127ba697f631 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:25:20.351821 4011 raft_consensus.cc:504] T 00000000000000000000000000000000 P a30949655f674e35942f127ba697f631 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a30949655f674e35942f127ba697f631" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46285 } }
I20260620 04:25:20.352175 4011 leader_election.cc:302] T 00000000000000000000000000000000 P a30949655f674e35942f127ba697f631 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a30949655f674e35942f127ba697f631; no voters:
I20260620 04:25:20.352461 4011 leader_election.cc:288] T 00000000000000000000000000000000 P a30949655f674e35942f127ba697f631 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:25:20.352566 4016 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a30949655f674e35942f127ba697f631 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:25:20.352759 4016 raft_consensus.cc:686] T 00000000000000000000000000000000 P a30949655f674e35942f127ba697f631 [term 1 LEADER]: Becoming Leader. State: Replica: a30949655f674e35942f127ba697f631, State: Running, Role: LEADER
I20260620 04:25:20.353135 4016 consensus_queue.cc:237] T 00000000000000000000000000000000 P a30949655f674e35942f127ba697f631 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a30949655f674e35942f127ba697f631" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46285 } }
I20260620 04:25:20.353542 4011 sys_catalog.cc:564] T 00000000000000000000000000000000 P a30949655f674e35942f127ba697f631 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:25:20.354591 4018 sys_catalog.cc:455] T 00000000000000000000000000000000 P a30949655f674e35942f127ba697f631 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a30949655f674e35942f127ba697f631. Latest consensus state: current_term: 1 leader_uuid: "a30949655f674e35942f127ba697f631" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a30949655f674e35942f127ba697f631" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46285 } } }
I20260620 04:25:20.354717 4018 sys_catalog.cc:458] T 00000000000000000000000000000000 P a30949655f674e35942f127ba697f631 [sys.catalog]: This master's current role is: LEADER
I20260620 04:25:20.354955 4017 sys_catalog.cc:455] T 00000000000000000000000000000000 P a30949655f674e35942f127ba697f631 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a30949655f674e35942f127ba697f631" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a30949655f674e35942f127ba697f631" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46285 } } }
I20260620 04:25:20.355029 4017 sys_catalog.cc:458] T 00000000000000000000000000000000 P a30949655f674e35942f127ba697f631 [sys.catalog]: This master's current role is: LEADER
W20260620 04:25:20.358068 4031 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a30949655f674e35942f127ba697f631: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 04:25:20.358140 4031 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 04:25:20.358263 4025 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:25:20.359120 4025 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:25:20.364581 4025 catalog_manager.cc:1347] Generated new cluster ID: 8ca2c3420b3143c1aaded5db40e4670e
I20260620 04:25:20.364648 4025 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:25:20.395635 4025 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:25:20.396682 4025 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:25:20.401413 4025 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a30949655f674e35942f127ba697f631: Generated new TSK 0
I20260620 04:25:20.402159 4025 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929496330398-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929496330398-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 04:25:20.430243 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929496330398-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929496330398-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929496330398-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929496330398-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46285
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929496330398-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:25:20.555891 4037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:25:20.556176 4037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:25:20.556223 4037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:25:20.560257 4037 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:25:20.560313 4037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:25:20.560382 4037 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 04:25:20.565585 4037 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929496330398-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929496330398-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929496330398-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929496330398-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46285
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4037
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929496330398-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:25:20.566696 4037 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:25:20.567665 4037 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:25:20.577078 4042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:25:20.577080 4045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:25:20.577394 4037 server_base.cc:1034] running on GCE node
W20260620 04:25:20.577405 4043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:25:20.577910 4037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:25:20.578477 4037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:25:20.579653 4037 hybrid_clock.cc:648] HybridClock initialized: now 1781929520579637 us; error 24 us; skew 500 ppm
Jun 20 04:25:20 dist-test-slave-nk6d krb5kdc[3930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929520, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:25:20.582492 4037 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 04:25:20.583367 4037 webserver.cc:460] Webserver started at http://127.0.98.1:33187/ using document root <none> and password file <none>
I20260620 04:25:20.583875 4037 fs_manager.cc:359] Metadata directory not provided
I20260620 04:25:20.583930 4037 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:25:20.584153 4037 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:25:20.586158 4037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929496330398-392-0/minicluster-data/ts-0/data/instance:
uuid: "dec476fae417420e9a0c7de665d30cdb"
format_stamp: "Formatted at 2026-06-20 04:25:20 on dist-test-slave-nk6d"
server_key: "66411e459cd5486b78e905184e242c22"
server_key_iv: "a9b6d2c579fc110c4f097248e5616f94"
server_key_version: "encryptionkey@0"
I20260620 04:25:20.586671 4037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929496330398-392-0/minicluster-data/ts-0/wal/instance:
uuid: "dec476fae417420e9a0c7de665d30cdb"
format_stamp: "Formatted at 2026-06-20 04:25:20 on dist-test-slave-nk6d"
server_key: "66411e459cd5486b78e905184e242c22"
server_key_iv: "a9b6d2c579fc110c4f097248e5616f94"
server_key_version: "encryptionkey@0"
I20260620 04:25:20.590364 4037 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 04:25:20.592864 4052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:20.594036 4037 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:25:20.594185 4037 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929496330398-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929496330398-392-0/minicluster-data/ts-0/wal
uuid: "dec476fae417420e9a0c7de665d30cdb"
format_stamp: "Formatted at 2026-06-20 04:25:20 on dist-test-slave-nk6d"
server_key: "66411e459cd5486b78e905184e242c22"
server_key_iv: "a9b6d2c579fc110c4f097248e5616f94"
server_key_version: "encryptionkey@0"
I20260620 04:25:20.594293 4037 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929496330398-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929496330398-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929496330398-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:25:20.622665 4037 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:25:20.626121 4037 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:25:20.626297 4037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:25:20.626883 4037 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:25:20.627707 4037 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:25:20.627751 4037 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:20.627794 4037 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:25:20.627822 4037 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:20.636425 4037 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41237
I20260620 04:25:20.636446 4165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41237 every 8 connection(s)
I20260620 04:25:20.637293 4037 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929496330398-392-0/minicluster-data/ts-0/data/info.pb
Jun 20 04:25:20 dist-test-slave-nk6d krb5kdc[3930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929520, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:25:20.647015 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 4037
I20260620 04:25:20.647135 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929496330398-392-0/minicluster-data/ts-0/wal/instance
I20260620 04:25:20.647419 392 external_mini_cluster.cc:1428] Setting key 4c6b346fb6ff624152c32f32640e0608
I20260620 04:25:20.651387 4014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:20.639088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48739 (local address 127.0.98.62:46285)
0620 04:25:20.639228 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:20.639231 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:20.640064 (+ 833us) server_negotiation.cc:407] Connection header received
0620 04:25:20.641460 (+ 1396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:20.641463 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:20.641507 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:20.641569 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:20.642749 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:20.643273 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:20.644520 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:20.644752 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:20.644832 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:20.647472 (+ 2640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:20.647476 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:20.647479 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:25:20.647509 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:20.649271 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:20.649765 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:20.649769 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:20.649771 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:20.649858 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:20.650147 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:20.650150 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:20.650151 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:20.650355 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:20.650454 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:20.651162 (+ 708us) server_negotiation.cc:299] Negotiation successful
0620 04:25:20.651270 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 04:25:20.652591 4168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:20.639472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46285 (local address 127.0.98.1:48739)
0620 04:25:20.639908 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:20.639974 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:20.641247 (+ 1273us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:20.641737 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:20.641744 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:20.642110 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:20.642617 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:20.642631 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:20.643396 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:20.643399 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:20.644412 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:20.644422 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:20.644889 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:20.644892 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:20.645058 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:20.645666 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:20.645684 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:20.647298 (+ 1614us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:25:20.649392 (+ 2094us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:20.649397 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:20.649411 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:20.649670 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:20.649939 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:20.649942 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:20.649943 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:20.650071 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:20.650490 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:25:20.650498 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:25:20.650852 (+ 354us) client_negotiation.cc:769] Sending connection context
0620 04:25:20.651089 (+ 237us) client_negotiation.cc:240] Negotiation successful
0620 04:25:20.651469 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":98,"threads_started":1}
I20260620 04:25:20.653772 4166 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46285
I20260620 04:25:20.654016 4166 heartbeater.cc:464] Registering TS with master...
I20260620 04:25:20.654510 4166 heartbeater.cc:511] Master 127.0.98.62:46285 requested a full tablet report, sending...
I20260620 04:25:20.655998 3975 ts_manager.cc:194] Registered new tserver with Master: dec476fae417420e9a0c7de665d30cdb (127.0.98.1:41237)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 04:25:20.657161 3975 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:48739
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929496330398-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929496330398-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 04:25:20.672950 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929496330398-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929496330398-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929496330398-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929496330398-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46285
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929496330398-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:25:20.782130 4171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:25:20.782368 4171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:25:20.782429 4171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:25:20.786420 4171 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:25:20.786478 4171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:25:20.786542 4171 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 04:25:20.791733 4171 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929496330398-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929496330398-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929496330398-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929496330398-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46285
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4171
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929496330398-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:25:20.792762 4171 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:25:20.793728 4171 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:25:20.802922 4179 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:25:20.802942 4177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:25:20.802922 4176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:25:20.803382 4171 server_base.cc:1034] running on GCE node
I20260620 04:25:20.803745 4171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:25:20.804383 4171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:25:20.805537 4171 hybrid_clock.cc:648] HybridClock initialized: now 1781929520805523 us; error 26 us; skew 500 ppm
Jun 20 04:25:20 dist-test-slave-nk6d krb5kdc[3930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929520, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:25:20.808270 4171 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 04:25:20.809118 4171 webserver.cc:460] Webserver started at http://127.0.98.2:36573/ using document root <none> and password file <none>
I20260620 04:25:20.809633 4171 fs_manager.cc:359] Metadata directory not provided
I20260620 04:25:20.809690 4171 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:25:20.809870 4171 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:25:20.811839 4171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929496330398-392-0/minicluster-data/ts-1/data/instance:
uuid: "1a73194520b64fc8be0d20435139cbc5"
format_stamp: "Formatted at 2026-06-20 04:25:20 on dist-test-slave-nk6d"
server_key: "883ac0d18c83b183da7ce7b47779cba3"
server_key_iv: "78acd2cccf2d29c0aac9bf3f24b8dbc4"
server_key_version: "encryptionkey@0"
I20260620 04:25:20.812412 4171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929496330398-392-0/minicluster-data/ts-1/wal/instance:
uuid: "1a73194520b64fc8be0d20435139cbc5"
format_stamp: "Formatted at 2026-06-20 04:25:20 on dist-test-slave-nk6d"
server_key: "883ac0d18c83b183da7ce7b47779cba3"
server_key_iv: "78acd2cccf2d29c0aac9bf3f24b8dbc4"
server_key_version: "encryptionkey@0"
I20260620 04:25:20.815987 4171 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:25:20.818411 4186 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:20.819587 4171 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:25:20.819689 4171 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929496330398-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929496330398-392-0/minicluster-data/ts-1/wal
uuid: "1a73194520b64fc8be0d20435139cbc5"
format_stamp: "Formatted at 2026-06-20 04:25:20 on dist-test-slave-nk6d"
server_key: "883ac0d18c83b183da7ce7b47779cba3"
server_key_iv: "78acd2cccf2d29c0aac9bf3f24b8dbc4"
server_key_version: "encryptionkey@0"
I20260620 04:25:20.819782 4171 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929496330398-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929496330398-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929496330398-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:25:20.843043 4171 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:25:20.846235 4171 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:25:20.846410 4171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:25:20.846987 4171 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:25:20.847889 4171 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:25:20.847935 4171 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:20.848011 4171 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:25:20.848039 4171 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:20.856566 4171 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42915
I20260620 04:25:20.856597 4299 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42915 every 8 connection(s)
I20260620 04:25:20.857443 4171 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929496330398-392-0/minicluster-data/ts-1/data/info.pb
I20260620 04:25:20.858948 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 4171
I20260620 04:25:20.859053 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929496330398-392-0/minicluster-data/ts-1/wal/instance
I20260620 04:25:20.859356 392 external_mini_cluster.cc:1428] Setting key a210eafba6a99ba9f056cd9e5d53e189
Jun 20 04:25:20 dist-test-slave-nk6d krb5kdc[3930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929520, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260620 04:25:20.871544 4014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:20.859168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41205 (local address 127.0.98.62:46285)
0620 04:25:20.859340 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:20.859343 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:20.860396 (+ 1053us) server_negotiation.cc:407] Connection header received
0620 04:25:20.861847 (+ 1451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:20.861850 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:20.861902 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:20.861980 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:20.863429 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:20.864212 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:20.865301 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:20.865638 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:20.865746 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:20.868187 (+ 2441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:20.868191 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:20.868194 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:25:20.868220 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:20.869551 (+ 1331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:20.870073 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:20.870075 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:20.870076 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:20.870152 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:20.870465 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:20.870469 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:20.870471 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:20.870685 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:20.870778 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:20.871267 (+ 489us) server_negotiation.cc:299] Negotiation successful
0620 04:25:20.871384 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260620 04:25:20.872411 4302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:20.859688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46285 (local address 127.0.98.2:41205)
0620 04:25:20.860187 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:20.860292 (+ 105us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:20.861651 (+ 1359us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:20.862143 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:20.862152 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:20.862619 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:20.863284 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:20.863303 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:20.864315 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:20.864317 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:20.865181 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:20.865190 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:20.865763 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:20.865765 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:20.865909 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:20.866438 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:20.866455 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:20.868028 (+ 1573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:25:20.869655 (+ 1627us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:20.869660 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:20.869673 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:20.869993 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:20.870234 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:20.870237 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:20.870238 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:20.870363 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:20.870793 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:25:20.870801 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:25:20.871044 (+ 243us) client_negotiation.cc:769] Sending connection context
0620 04:25:20.871215 (+ 171us) client_negotiation.cc:240] Negotiation successful
0620 04:25:20.871533 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":107,"threads_started":1}
I20260620 04:25:20.873600 4300 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46285
I20260620 04:25:20.873839 4300 heartbeater.cc:464] Registering TS with master...
I20260620 04:25:20.874310 4300 heartbeater.cc:511] Master 127.0.98.62:46285 requested a full tablet report, sending...
I20260620 04:25:20.875339 3975 ts_manager.cc:194] Registered new tserver with Master: 1a73194520b64fc8be0d20435139cbc5 (127.0.98.2:42915)
I20260620 04:25:20.876132 3975 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41205
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929496330398-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929496330398-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 04:25:20.884595 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929496330398-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929496330398-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929496330398-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929496330398-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46285
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929496330398-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:25:20.992491 4305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:25:20.992764 4305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:25:20.992810 4305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:25:20.996749 4305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:25:20.996814 4305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:25:20.996876 4305 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 04:25:21.002214 4305 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929496330398-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929496330398-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929496330398-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929496330398-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46285
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4305
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929496330398-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:25:21.003198 4305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:25:21.004161 4305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:25:21.013762 4311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:25:21.013808 4305 server_base.cc:1034] running on GCE node
W20260620 04:25:21.013789 4310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:25:21.013759 4313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:25:21.014391 4305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:25:21.015008 4305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:25:21.016165 4305 hybrid_clock.cc:648] HybridClock initialized: now 1781929521016149 us; error 28 us; skew 500 ppm
Jun 20 04:25:21 dist-test-slave-nk6d krb5kdc[3930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929521, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:25:21.018819 4305 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 04:25:21.019680 4305 webserver.cc:460] Webserver started at http://127.0.98.3:35379/ using document root <none> and password file <none>
I20260620 04:25:21.020238 4305 fs_manager.cc:359] Metadata directory not provided
I20260620 04:25:21.020304 4305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:25:21.020498 4305 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:25:21.022490 4305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929496330398-392-0/minicluster-data/ts-2/data/instance:
uuid: "700afe59a5d046bea03ef37b518d877b"
format_stamp: "Formatted at 2026-06-20 04:25:21 on dist-test-slave-nk6d"
server_key: "34b890c2bbb606b95b842bdbf78328e3"
server_key_iv: "5d9540f9f00494394378faa33f3a02c8"
server_key_version: "encryptionkey@0"
I20260620 04:25:21.023062 4305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929496330398-392-0/minicluster-data/ts-2/wal/instance:
uuid: "700afe59a5d046bea03ef37b518d877b"
format_stamp: "Formatted at 2026-06-20 04:25:21 on dist-test-slave-nk6d"
server_key: "34b890c2bbb606b95b842bdbf78328e3"
server_key_iv: "5d9540f9f00494394378faa33f3a02c8"
server_key_version: "encryptionkey@0"
I20260620 04:25:21.026713 4305 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260620 04:25:21.029134 4320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:21.030349 4305 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 04:25:21.030450 4305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929496330398-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929496330398-392-0/minicluster-data/ts-2/wal
uuid: "700afe59a5d046bea03ef37b518d877b"
format_stamp: "Formatted at 2026-06-20 04:25:21 on dist-test-slave-nk6d"
server_key: "34b890c2bbb606b95b842bdbf78328e3"
server_key_iv: "5d9540f9f00494394378faa33f3a02c8"
server_key_version: "encryptionkey@0"
I20260620 04:25:21.030535 4305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929496330398-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929496330398-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929496330398-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:25:21.046577 4305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:25:21.049641 4305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:25:21.049820 4305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:25:21.050381 4305 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:25:21.051157 4305 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:25:21.051199 4305 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:21.051251 4305 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:25:21.051280 4305 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:21.059580 4305 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43975
I20260620 04:25:21.059696 4433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43975 every 8 connection(s)
I20260620 04:25:21.060496 4305 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929496330398-392-0/minicluster-data/ts-2/data/info.pb
Jun 20 04:25:21 dist-test-slave-nk6d krb5kdc[3930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929521, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:25:21.070505 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 4305
I20260620 04:25:21.070626 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929496330398-392-0/minicluster-data/ts-2/wal/instance
I20260620 04:25:21.070926 392 external_mini_cluster.cc:1428] Setting key 1e92bae8919c2c9371ae01f1dda902c9
I20260620 04:25:21.074054 4014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:21.062234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36723 (local address 127.0.98.62:46285)
0620 04:25:21.062355 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:21.062358 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:21.063186 (+ 828us) server_negotiation.cc:407] Connection header received
0620 04:25:21.064578 (+ 1392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:21.064581 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:21.064635 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:21.064709 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:21.065884 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:21.066397 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:21.067480 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:21.067688 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:21.067772 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:21.070100 (+ 2328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:21.070104 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:21.070106 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:25:21.070131 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:21.071708 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:21.072314 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:21.072317 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:21.072318 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:21.072407 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:21.072784 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:21.072788 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:21.072790 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:21.073028 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:21.073194 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:21.073737 (+ 543us) server_negotiation.cc:299] Negotiation successful
0620 04:25:21.073876 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 04:25:21.074853 4436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:21.062610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46285 (local address 127.0.98.3:36723)
0620 04:25:21.063043 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:21.063085 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:21.064379 (+ 1294us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:21.064895 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:21.064901 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:21.065260 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:21.065733 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:21.065747 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:21.066507 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:21.066510 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:21.067367 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:21.067376 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:21.067819 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:21.067823 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:21.068009 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:21.068520 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:21.068536 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:21.069937 (+ 1401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:25:21.071835 (+ 1898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:21.071841 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:21.071857 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:21.072203 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:21.072506 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:21.072509 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:21.072512 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:21.072677 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:21.073214 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:25:21.073222 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:25:21.073473 (+ 251us) client_negotiation.cc:769] Sending connection context
0620 04:25:21.073664 (+ 191us) client_negotiation.cc:240] Negotiation successful
0620 04:25:21.074016 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":104,"threads_started":1}
I20260620 04:25:21.075932 4434 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46285
I20260620 04:25:21.076202 4434 heartbeater.cc:464] Registering TS with master...
I20260620 04:25:21.076700 4434 heartbeater.cc:511] Master 127.0.98.62:46285 requested a full tablet report, sending...
I20260620 04:25:21.077723 3975 ts_manager.cc:194] Registered new tserver with Master: 700afe59a5d046bea03ef37b518d877b (127.0.98.3:43975)
I20260620 04:25:21.078287 3975 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:36723
I20260620 04:25:21.084779 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 04:25:21 dist-test-slave-nk6d krb5kdc[3930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929521, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 04:25:21 dist-test-slave-nk6d krb5kdc[3930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929521, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 04:25:21.108879 4441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:21.099742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54594 (local address 127.0.98.1:41237)
0620 04:25:21.100041 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:21.100056 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:21.100082 (+ 26us) server_negotiation.cc:407] Connection header received
0620 04:25:21.100233 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:21.100240 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:21.100436 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:21.100569 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:21.101271 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:21.102062 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:21.102880 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:21.103257 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:21.103352 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:21.105643 (+ 2291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:21.105651 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:21.105664 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:25:21.105688 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:21.107183 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:21.107590 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:21.107596 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:21.107597 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:21.107678 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:21.107878 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:21.107880 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:21.107881 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:21.108150 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:21.108402 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:21.108557 (+ 155us) server_negotiation.cc:299] Negotiation successful
0620 04:25:21.108703 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":65,"threads_started":1}
Jun 20 04:25:21 dist-test-slave-nk6d krb5kdc[3930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929521, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:25:21.118659 4014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:21.111524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40952 (local address 127.0.98.62:46285)
0620 04:25:21.111635 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:21.111638 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:21.111697 (+ 59us) server_negotiation.cc:407] Connection header received
0620 04:25:21.111784 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:21.111786 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:21.111825 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:21.111888 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:21.112444 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:21.112934 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:21.113713 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:21.113912 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:21.113975 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:21.116242 (+ 2267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:21.116245 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:21.116247 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:25:21.116270 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:21.117385 (+ 1115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:21.117726 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:21.117728 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:21.117729 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:21.117793 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:21.118026 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:21.118028 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:21.118029 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:21.118213 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:21.118330 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:21.118460 (+ 130us) server_negotiation.cc:299] Negotiation successful
0620 04:25:21.118546 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260620 04:25:21.119024 3975 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40952
I20260620 04:25:21.119537 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 4037
I20260620 04:25:21.124284 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 4171
I20260620 04:25:21.128793 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 4305
I20260620 04:25:21.133096 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 3943
2026-06-20T04:25:21Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2050 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929496330398-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:25:21 dist-test-slave-nk6d krb5kdc[4446](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:25:21 dist-test-slave-nk6d krb5kdc[4446](info): set up 2 sockets
Jun 20 04:25:21 dist-test-slave-nk6d krb5kdc[4446](info): commencing operation
krb5kdc: starting...
W20260620 04:25:22.071873 392 mini_kdc.cc:121] Time spent starting KDC: real 0.919s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929496330398-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929496330398-392-0/krb5kdc/test-user.keytab.
Jun 20 04:25:22 dist-test-slave-nk6d krb5kdc[4446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929522, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:25:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:25:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 04:25:22.191766 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929496330398-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929496330398-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929496330398-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41517
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41517
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:25:22.296463 4459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:25:22.296726 4459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:25:22.296768 4459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:25:22.300839 4459 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:25:22.300901 4459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:25:22.300920 4459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:25:22.300940 4459 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:25:22.300954 4459 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:25:22.306200 4459 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929496330398-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929496330398-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41517
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41517
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4459
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929496330398-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:25:22.307255 4459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:25:22.308274 4459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 04:25:22.316330 4459 server_base.cc:1034] running on GCE node
W20260620 04:25:22.316282 4465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:25:22.316262 4464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:25:22.316296 4467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:25:22.317152 4459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:25:22.318183 4459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:25:22.319332 4459 hybrid_clock.cc:648] HybridClock initialized: now 1781929522319298 us; error 53 us; skew 500 ppm
Jun 20 04:25:22 dist-test-slave-nk6d krb5kdc[4446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929522, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:25:22.321959 4459 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 04:25:22.322641 4459 fs_manager.cc:359] Metadata directory not provided
I20260620 04:25:22.322703 4459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:25:22.322887 4459 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:25:22.324853 4459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929496330398-392-0/minicluster-data/master-0/data/instance:
uuid: "fdf38559144d41bfb61537a0db395043"
format_stamp: "Formatted at 2026-06-20 04:25:22 on dist-test-slave-nk6d"
server_key: "bec5fb06159261f58f354f64b4cf9308"
server_key_iv: "cd9befb9adc3ffe48bfead9a82344447"
server_key_version: "encryptionkey@0"
I20260620 04:25:22.325382 4459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929496330398-392-0/minicluster-data/master-0/wal/instance:
uuid: "fdf38559144d41bfb61537a0db395043"
format_stamp: "Formatted at 2026-06-20 04:25:22 on dist-test-slave-nk6d"
server_key: "bec5fb06159261f58f354f64b4cf9308"
server_key_iv: "cd9befb9adc3ffe48bfead9a82344447"
server_key_version: "encryptionkey@0"
I20260620 04:25:22.328914 4459 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 04:25:22.331219 4473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:22.332453 4459 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:25:22.332558 4459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929496330398-392-0/minicluster-data/master-0/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929496330398-392-0/minicluster-data/master-0/wal
uuid: "fdf38559144d41bfb61537a0db395043"
format_stamp: "Formatted at 2026-06-20 04:25:22 on dist-test-slave-nk6d"
server_key: "bec5fb06159261f58f354f64b4cf9308"
server_key_iv: "cd9befb9adc3ffe48bfead9a82344447"
server_key_version: "encryptionkey@0"
I20260620 04:25:22.332647 4459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929496330398-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929496330398-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929496330398-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:25:22.344592 4459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:25:22.347574 4459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:25:22.347734 4459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:25:22.354128 4459 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41517
I20260620 04:25:22.354367 4525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41517 every 8 connection(s)
I20260620 04:25:22.354722 4459 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:25:22.357237 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 4459
I20260620 04:25:22.357329 4526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:25:22.357350 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929496330398-392-0/minicluster-data/master-0/wal/instance
I20260620 04:25:22.357611 392 external_mini_cluster.cc:1428] Setting key 94efd12c3fb84bdfa51f654e9ee5b922
I20260620 04:25:22.364267 4526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdf38559144d41bfb61537a0db395043: Bootstrap starting.
Jun 20 04:25:22 dist-test-slave-nk6d krb5kdc[4446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929522, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:25:22.366577 4526 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fdf38559144d41bfb61537a0db395043: Neither blocks nor log segments found. Creating new log.
I20260620 04:25:22.367326 4526 log.cc:824] T 00000000000000000000000000000000 P fdf38559144d41bfb61537a0db395043: Log is configured to *not* fsync() on all Append() calls
I20260620 04:25:22.369639 4526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdf38559144d41bfb61537a0db395043: No bootstrap required, opened a new log
I20260620 04:25:22.372104 4529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:22.358870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35254 (local address 127.0.98.62:41517)
0620 04:25:22.359297 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:22.359315 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:22.359359 (+ 44us) server_negotiation.cc:407] Connection header received
0620 04:25:22.360036 (+ 677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:22.360060 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:22.360494 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:22.360863 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:22.361496 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:22.362913 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:22.363832 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:22.364537 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:22.364690 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:22.367131 (+ 2441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:22.367137 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:22.367153 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:25:22.367185 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:22.369101 (+ 1916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:22.369512 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:22.369517 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:22.369524 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:22.369664 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:22.369926 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:22.369937 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:22.369939 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:22.370424 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:22.370597 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:22.370788 (+ 191us) server_negotiation.cc:299] Negotiation successful
0620 04:25:22.371091 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":113,"threads_started":1}
I20260620 04:25:22.373868 4526 raft_consensus.cc:348] T 00000000000000000000000000000000 P fdf38559144d41bfb61537a0db395043 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdf38559144d41bfb61537a0db395043" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41517 } }
I20260620 04:25:22.374135 4526 raft_consensus.cc:374] T 00000000000000000000000000000000 P fdf38559144d41bfb61537a0db395043 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:25:22.374194 4526 raft_consensus.cc:729] T 00000000000000000000000000000000 P fdf38559144d41bfb61537a0db395043 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdf38559144d41bfb61537a0db395043, State: Initialized, Role: FOLLOWER
I20260620 04:25:22.374606 4526 consensus_queue.cc:260] T 00000000000000000000000000000000 P fdf38559144d41bfb61537a0db395043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdf38559144d41bfb61537a0db395043" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41517 } }
I20260620 04:25:22.374732 4526 raft_consensus.cc:388] T 00000000000000000000000000000000 P fdf38559144d41bfb61537a0db395043 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:25:22.374794 4526 raft_consensus.cc:482] T 00000000000000000000000000000000 P fdf38559144d41bfb61537a0db395043 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:25:22.374888 4526 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fdf38559144d41bfb61537a0db395043 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:25:22.375748 4526 raft_consensus.cc:504] T 00000000000000000000000000000000 P fdf38559144d41bfb61537a0db395043 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdf38559144d41bfb61537a0db395043" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41517 } }
I20260620 04:25:22.376102 4526 leader_election.cc:302] T 00000000000000000000000000000000 P fdf38559144d41bfb61537a0db395043 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fdf38559144d41bfb61537a0db395043; no voters:
I20260620 04:25:22.376405 4526 leader_election.cc:288] T 00000000000000000000000000000000 P fdf38559144d41bfb61537a0db395043 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:25:22.376533 4531 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fdf38559144d41bfb61537a0db395043 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:25:22.376726 4531 raft_consensus.cc:686] T 00000000000000000000000000000000 P fdf38559144d41bfb61537a0db395043 [term 1 LEADER]: Becoming Leader. State: Replica: fdf38559144d41bfb61537a0db395043, State: Running, Role: LEADER
I20260620 04:25:22.377067 4531 consensus_queue.cc:237] T 00000000000000000000000000000000 P fdf38559144d41bfb61537a0db395043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdf38559144d41bfb61537a0db395043" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41517 } }
I20260620 04:25:22.377537 4526 sys_catalog.cc:564] T 00000000000000000000000000000000 P fdf38559144d41bfb61537a0db395043 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:25:22.378657 4533 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdf38559144d41bfb61537a0db395043 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fdf38559144d41bfb61537a0db395043. Latest consensus state: current_term: 1 leader_uuid: "fdf38559144d41bfb61537a0db395043" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdf38559144d41bfb61537a0db395043" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41517 } } }
I20260620 04:25:22.378695 4532 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdf38559144d41bfb61537a0db395043 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fdf38559144d41bfb61537a0db395043" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdf38559144d41bfb61537a0db395043" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41517 } } }
I20260620 04:25:22.378780 4533 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdf38559144d41bfb61537a0db395043 [sys.catalog]: This master's current role is: LEADER
I20260620 04:25:22.378789 4532 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdf38559144d41bfb61537a0db395043 [sys.catalog]: This master's current role is: LEADER
I20260620 04:25:22.379155 4540 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:25:22.382032 4540 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:25:22.386880 4540 catalog_manager.cc:1347] Generated new cluster ID: 084c8d4f513a4d9db7a66419a5a459f2
I20260620 04:25:22.386943 4540 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:25:22.404420 4540 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:25:22.405712 4540 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:25:22.423606 4540 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fdf38559144d41bfb61537a0db395043: Generated new TSK 0
I20260620 04:25:22.424525 4540 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929496330398-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929496330398-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 04:25:22.453004 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929496330398-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929496330398-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929496330398-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929496330398-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41517
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929496330398-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:25:22.561410 4552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:25:22.561669 4552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:25:22.561714 4552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:25:22.565680 4552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:25:22.565743 4552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:25:22.565812 4552 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 04:25:22.571141 4552 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929496330398-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929496330398-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929496330398-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929496330398-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4552
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929496330398-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:25:22.572221 4552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:25:22.573312 4552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:25:22.582475 4558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:25:22.582604 4560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:25:22.582484 4557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:25:22.582664 4552 server_base.cc:1034] running on GCE node
I20260620 04:25:22.583302 4552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:25:22.583985 4552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:25:22.585152 4552 hybrid_clock.cc:648] HybridClock initialized: now 1781929522585130 us; error 31 us; skew 500 ppm
Jun 20 04:25:22 dist-test-slave-nk6d krb5kdc[4446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929522, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:25:22.587798 4552 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 04:25:22.588557 4552 fs_manager.cc:359] Metadata directory not provided
I20260620 04:25:22.588614 4552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:25:22.588826 4552 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:25:22.590809 4552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929496330398-392-0/minicluster-data/ts-0/data/instance:
uuid: "a2c69b44b06042c39176ed40803f060b"
format_stamp: "Formatted at 2026-06-20 04:25:22 on dist-test-slave-nk6d"
server_key: "d64bf8eae97999d2ea8d30b01b2668ca"
server_key_iv: "c45a3dccb73833d7af3dead7eb3a9a37"
server_key_version: "encryptionkey@0"
I20260620 04:25:22.591300 4552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929496330398-392-0/minicluster-data/ts-0/wal/instance:
uuid: "a2c69b44b06042c39176ed40803f060b"
format_stamp: "Formatted at 2026-06-20 04:25:22 on dist-test-slave-nk6d"
server_key: "d64bf8eae97999d2ea8d30b01b2668ca"
server_key_iv: "c45a3dccb73833d7af3dead7eb3a9a37"
server_key_version: "encryptionkey@0"
I20260620 04:25:22.594727 4552 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260620 04:25:22.597060 4566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:22.598304 4552 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 04:25:22.598408 4552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929496330398-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929496330398-392-0/minicluster-data/ts-0/wal
uuid: "a2c69b44b06042c39176ed40803f060b"
format_stamp: "Formatted at 2026-06-20 04:25:22 on dist-test-slave-nk6d"
server_key: "d64bf8eae97999d2ea8d30b01b2668ca"
server_key_iv: "c45a3dccb73833d7af3dead7eb3a9a37"
server_key_version: "encryptionkey@0"
I20260620 04:25:22.598513 4552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929496330398-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929496330398-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929496330398-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:25:22.610183 4552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:25:22.613052 4552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:25:22.613215 4552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:25:22.613744 4552 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:25:22.614527 4552 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:25:22.614573 4552 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:22.614610 4552 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:25:22.614636 4552 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:22.622720 4552 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40965
I20260620 04:25:22.622740 4679 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40965 every 8 connection(s)
I20260620 04:25:22.623298 4552 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929496330398-392-0/minicluster-data/ts-0/data/info.pb
I20260620 04:25:22.628988 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 4552
I20260620 04:25:22.629088 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929496330398-392-0/minicluster-data/ts-0/wal/instance
I20260620 04:25:22.629335 392 external_mini_cluster.cc:1428] Setting key fc61d2c0c353b3f8c0a71a9a310c42e0
Jun 20 04:25:22 dist-test-slave-nk6d krb5kdc[4446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929522, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:25:22.636859 4529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:22.624904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35537 (local address 127.0.98.62:41517)
0620 04:25:22.625036 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:22.625040 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:22.625847 (+ 807us) server_negotiation.cc:407] Connection header received
0620 04:25:22.627149 (+ 1302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:22.627152 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:22.627208 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:22.627345 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:22.628508 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:22.629035 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:22.630391 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:22.630699 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:22.630801 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:22.633128 (+ 2327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:22.633131 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:22.633134 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:25:22.633159 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:22.634616 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:22.635160 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:22.635164 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:22.635166 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:22.635283 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:22.635665 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:22.635668 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:22.635670 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:22.635898 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:22.636043 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:22.636566 (+ 523us) server_negotiation.cc:299] Negotiation successful
0620 04:25:22.636696 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 04:25:22.637707 4682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:22.625278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41517 (local address 127.0.98.1:35537)
0620 04:25:22.625692 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:22.625732 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:22.626920 (+ 1188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:22.627490 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:22.627496 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:22.627847 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:22.628363 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:22.628377 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:22.629200 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:22.629202 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:22.630294 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:22.630304 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:22.630835 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:22.630838 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:22.631018 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:22.631587 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:22.631603 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:22.632979 (+ 1376us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:25:22.634742 (+ 1763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:22.634747 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:22.634759 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:22.635056 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:22.635399 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:22.635402 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:22.635404 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:22.635563 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:22.636047 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:25:22.636054 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:25:22.636295 (+ 241us) client_negotiation.cc:769] Sending connection context
0620 04:25:22.636499 (+ 204us) client_negotiation.cc:240] Negotiation successful
0620 04:25:22.636821 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":102,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 04:25:22.638886 4680 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41517
I20260620 04:25:22.639122 4680 heartbeater.cc:464] Registering TS with master...
I20260620 04:25:22.639544 4680 heartbeater.cc:511] Master 127.0.98.62:41517 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 04:25:22.641160 4490 ts_manager.cc:194] Registered new tserver with Master: a2c69b44b06042c39176ed40803f060b (127.0.98.1:40965)
I20260620 04:25:22.642311 4490 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:35537
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929496330398-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929496330398-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 04:25:22.652850 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929496330398-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929496330398-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929496330398-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929496330398-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41517
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929496330398-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:25:22.758940 4685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:25:22.759196 4685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:25:22.759238 4685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:25:22.763296 4685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:25:22.763357 4685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:25:22.763417 4685 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 04:25:22.768653 4685 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929496330398-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929496330398-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929496330398-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929496330398-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4685
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929496330398-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:25:22.769685 4685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:25:22.770649 4685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:25:22.779870 4693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:25:22.779881 4690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:25:22.780133 4691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:25:22.780422 4685 server_base.cc:1034] running on GCE node
I20260620 04:25:22.780778 4685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:25:22.781355 4685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:25:22.782524 4685 hybrid_clock.cc:648] HybridClock initialized: now 1781929522782502 us; error 32 us; skew 500 ppm
Jun 20 04:25:22 dist-test-slave-nk6d krb5kdc[4446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929522, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:25:22.785197 4685 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 04:25:22.785843 4685 fs_manager.cc:359] Metadata directory not provided
I20260620 04:25:22.785903 4685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:25:22.786077 4685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:25:22.788081 4685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929496330398-392-0/minicluster-data/ts-1/data/instance:
uuid: "b36a6e673aeb44338eaefc2db24c5280"
format_stamp: "Formatted at 2026-06-20 04:25:22 on dist-test-slave-nk6d"
server_key: "0683b01d98d1b4ea6345132c90745820"
server_key_iv: "6b14936ae20f9234819a132be3243a9c"
server_key_version: "encryptionkey@0"
I20260620 04:25:22.788589 4685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929496330398-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b36a6e673aeb44338eaefc2db24c5280"
format_stamp: "Formatted at 2026-06-20 04:25:22 on dist-test-slave-nk6d"
server_key: "0683b01d98d1b4ea6345132c90745820"
server_key_iv: "6b14936ae20f9234819a132be3243a9c"
server_key_version: "encryptionkey@0"
I20260620 04:25:22.792038 4685 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 04:25:22.794225 4699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:22.795454 4685 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:25:22.795548 4685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929496330398-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929496330398-392-0/minicluster-data/ts-1/wal
uuid: "b36a6e673aeb44338eaefc2db24c5280"
format_stamp: "Formatted at 2026-06-20 04:25:22 on dist-test-slave-nk6d"
server_key: "0683b01d98d1b4ea6345132c90745820"
server_key_iv: "6b14936ae20f9234819a132be3243a9c"
server_key_version: "encryptionkey@0"
I20260620 04:25:22.795630 4685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929496330398-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929496330398-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929496330398-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:25:22.814500 4685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:25:22.817596 4685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:25:22.817745 4685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:25:22.818279 4685 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:25:22.819051 4685 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:25:22.819096 4685 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:22.819135 4685 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:25:22.819155 4685 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:22.827055 4685 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36043
I20260620 04:25:22.827113 4812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36043 every 8 connection(s)
I20260620 04:25:22.827606 4685 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929496330398-392-0/minicluster-data/ts-1/data/info.pb
I20260620 04:25:22.828398 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 4685
I20260620 04:25:22.828508 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929496330398-392-0/minicluster-data/ts-1/wal/instance
I20260620 04:25:22.828823 392 external_mini_cluster.cc:1428] Setting key 2ca99a37b2fb9ec0496f3906ba5e720a
Jun 20 04:25:22 dist-test-slave-nk6d krb5kdc[4446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929522, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260620 04:25:22.842104 4529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:22.832088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54823 (local address 127.0.98.62:41517)
0620 04:25:22.832219 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:22.832223 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:22.832239 (+ 16us) server_negotiation.cc:407] Connection header received
0620 04:25:22.832628 (+ 389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:22.832632 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:22.832682 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:22.832764 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:22.833843 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:22.834376 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:22.835607 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:22.835828 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:22.835932 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:22.838660 (+ 2728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:22.838663 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:22.838665 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:25:22.838689 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:22.840074 (+ 1385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:22.840551 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:22.840554 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:22.840555 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:22.840629 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:22.840933 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:22.840936 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:22.840937 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:22.841161 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:22.841245 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:22.841813 (+ 568us) server_negotiation.cc:299] Negotiation successful
0620 04:25:22.841930 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 04:25:22.842872 4815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:22.829853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41517 (local address 127.0.98.2:54823)
0620 04:25:22.830872 (+ 1019us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:22.830932 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:22.832442 (+ 1510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:22.832902 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:22.832909 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:22.833243 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:22.833719 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:22.833732 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:22.834500 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:22.834503 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:22.835467 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:22.835478 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:22.836001 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:22.836006 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:22.836211 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:22.836774 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:22.836793 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:22.838463 (+ 1670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:25:22.840183 (+ 1720us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:22.840187 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:22.840200 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:22.840436 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:22.840700 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:22.840702 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:22.840704 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:22.840828 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:22.841261 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:25:22.841267 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:25:22.841579 (+ 312us) client_negotiation.cc:769] Sending connection context
0620 04:25:22.841749 (+ 170us) client_negotiation.cc:240] Negotiation successful
0620 04:25:22.842069 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":648,"thread_start_us":130,"threads_started":1}
I20260620 04:25:22.844079 4813 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41517
I20260620 04:25:22.844327 4813 heartbeater.cc:464] Registering TS with master...
I20260620 04:25:22.844854 4813 heartbeater.cc:511] Master 127.0.98.62:41517 requested a full tablet report, sending...
I20260620 04:25:22.845940 4490 ts_manager.cc:194] Registered new tserver with Master: b36a6e673aeb44338eaefc2db24c5280 (127.0.98.2:36043)
I20260620 04:25:22.846715 4490 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:54823
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929496330398-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929496330398-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 04:25:22.855721 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929496330398-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929496330398-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929496330398-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929496330398-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41517
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929496330398-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:25:22.962013 4818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:25:22.962260 4818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:25:22.962301 4818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:25:22.966213 4818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:25:22.966272 4818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:25:22.966331 4818 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 04:25:22.971386 4818 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929496330398-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929496330398-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929496330398-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929496330398-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4818
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929496330398-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:25:22.972357 4818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:25:22.973296 4818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:25:22.982695 4823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:25:22.982822 4818 server_base.cc:1034] running on GCE node
W20260620 04:25:22.982695 4824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:25:22.982698 4826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:25:22.983350 4818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:25:22.983929 4818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:25:22.985098 4818 hybrid_clock.cc:648] HybridClock initialized: now 1781929522985073 us; error 38 us; skew 500 ppm
Jun 20 04:25:22 dist-test-slave-nk6d krb5kdc[4446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929522, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:25:22.987676 4818 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 04:25:22.988390 4818 fs_manager.cc:359] Metadata directory not provided
I20260620 04:25:22.988454 4818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:25:22.988633 4818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:25:22.990661 4818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929496330398-392-0/minicluster-data/ts-2/data/instance:
uuid: "e79b9c19c2cd453880e41e81b3e0eb9f"
format_stamp: "Formatted at 2026-06-20 04:25:22 on dist-test-slave-nk6d"
server_key: "a0f23335ccfaaf91eb51fff51e2c4b3e"
server_key_iv: "e90aa6aaca1818da852249059b9bb064"
server_key_version: "encryptionkey@0"
I20260620 04:25:22.991175 4818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929496330398-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e79b9c19c2cd453880e41e81b3e0eb9f"
format_stamp: "Formatted at 2026-06-20 04:25:22 on dist-test-slave-nk6d"
server_key: "a0f23335ccfaaf91eb51fff51e2c4b3e"
server_key_iv: "e90aa6aaca1818da852249059b9bb064"
server_key_version: "encryptionkey@0"
I20260620 04:25:22.994654 4818 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 04:25:22.996964 4832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:22.998173 4818 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:25:22.998281 4818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929496330398-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929496330398-392-0/minicluster-data/ts-2/wal
uuid: "e79b9c19c2cd453880e41e81b3e0eb9f"
format_stamp: "Formatted at 2026-06-20 04:25:22 on dist-test-slave-nk6d"
server_key: "a0f23335ccfaaf91eb51fff51e2c4b3e"
server_key_iv: "e90aa6aaca1818da852249059b9bb064"
server_key_version: "encryptionkey@0"
I20260620 04:25:22.998368 4818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929496330398-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929496330398-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929496330398-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:25:23.018814 4818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:25:23.021955 4818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:25:23.022135 4818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:25:23.022696 4818 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:25:23.023512 4818 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:25:23.023559 4818 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:23.023610 4818 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:25:23.023634 4818 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:23.031757 4818 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42851
I20260620 04:25:23.031780 4945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42851 every 8 connection(s)
I20260620 04:25:23.032346 4818 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929496330398-392-0/minicluster-data/ts-2/data/info.pb
Jun 20 04:25:23 dist-test-slave-nk6d krb5kdc[4446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929522, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:25:23.041311 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 4818
I20260620 04:25:23.041412 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929496330398-392-0/minicluster-data/ts-2/wal/instance
I20260620 04:25:23.041698 392 external_mini_cluster.cc:1428] Setting key 8ad8191fe6d085bbc17bd5df34066114
I20260620 04:25:23.044958 4529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:23.033964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42887 (local address 127.0.98.62:41517)
0620 04:25:23.034082 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:23.034085 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:23.034887 (+ 802us) server_negotiation.cc:407] Connection header received
0620 04:25:23.036284 (+ 1397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:23.036287 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:23.036343 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:23.036441 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:23.037596 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:23.038102 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:23.039152 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:23.039346 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:23.039409 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:23.041658 (+ 2249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:23.041662 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:23.041665 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:25:23.041692 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:23.043153 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:23.043688 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:23.043692 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:23.043693 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:23.043780 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:23.044103 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:23.044105 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:23.044106 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:23.044267 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:23.044337 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:23.044759 (+ 422us) server_negotiation.cc:299] Negotiation successful
0620 04:25:23.044847 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 04:25:23.045724 4948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:23.034330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41517 (local address 127.0.98.3:42887)
0620 04:25:23.034751 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:23.034791 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:23.036083 (+ 1292us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:23.036621 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:23.036628 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:23.036995 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:23.037471 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:23.037484 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:23.038189 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:23.038191 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:23.039064 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:23.039072 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:23.039445 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:23.039447 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:23.039580 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:23.040105 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:23.040120 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:23.041493 (+ 1373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:25:23.043250 (+ 1757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:23.043256 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:23.043269 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:23.043601 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:23.043851 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:23.043853 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:23.043854 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:23.044030 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:23.044337 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:25:23.044342 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:25:23.044569 (+ 227us) client_negotiation.cc:769] Sending connection context
0620 04:25:23.044714 (+ 145us) client_negotiation.cc:240] Negotiation successful
0620 04:25:23.044967 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":111,"threads_started":1}
I20260620 04:25:23.046687 4946 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41517
I20260620 04:25:23.046904 4946 heartbeater.cc:464] Registering TS with master...
I20260620 04:25:23.047329 4946 heartbeater.cc:511] Master 127.0.98.62:41517 requested a full tablet report, sending...
I20260620 04:25:23.048348 4490 ts_manager.cc:194] Registered new tserver with Master: e79b9c19c2cd453880e41e81b3e0eb9f (127.0.98.3:42851)
I20260620 04:25:23.048913 4490 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:42887
I20260620 04:25:23.055326 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 04:25:23.064174 4529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:23.057167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35264 (local address 127.0.98.62:41517)
0620 04:25:23.057277 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:23.057282 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:23.057419 (+ 137us) server_negotiation.cc:407] Connection header received
0620 04:25:23.057546 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:23.057550 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:23.057597 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:23.057676 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:23.058304 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:23.058816 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:23.059601 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:23.059822 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:23.059905 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:23.061213 (+ 1308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:23.061217 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:23.061221 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:25:23.061250 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:23.062622 (+ 1372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:23.063018 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:23.063020 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:23.063022 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:23.063094 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:23.063282 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:23.063284 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:23.063285 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:23.063490 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:23.063591 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:23.063832 (+ 241us) server_negotiation.cc:299] Negotiation successful
0620 04:25:23.064008 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 04:25:23.066823 4490 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35264:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 04:25:23.069808 4490 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 04:25:23.083485 4957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:23.077752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42851 (local address 127.0.0.1:53934)
0620 04:25:23.078130 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:23.078178 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:23.078539 (+ 361us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:23.079275 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:23.079282 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:23.079302 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:25:23.079422 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:23.079429 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:23.080825 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:23.080828 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:23.081977 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:23.081986 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:23.082996 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:23.082999 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:23.083115 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:23.083148 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 04:25:23.083242 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 04:25:23.083334 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"mutex_wait_us":42,"thread_start_us":79,"threads_started":1}
I20260620 04:25:23.083539 4958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:23.077904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53934 (local address 127.0.98.3:42851)
0620 04:25:23.078504 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:23.078517 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:23.078543 (+ 26us) server_negotiation.cc:407] Connection header received
0620 04:25:23.078738 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:23.078744 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:23.078990 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:23.079144 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:25:23.079544 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:23.080697 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:23.082104 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:23.082869 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:23.082976 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:23.083121 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:23.083285 (+ 164us) server_negotiation.cc:299] Negotiation successful
0620 04:25:23.083376 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":151,"threads_started":1}
I20260620 04:25:23.083809 4960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:23.078144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36043 (local address 127.0.0.1:38260)
0620 04:25:23.078787 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:23.078800 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:23.078882 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:23.080062 (+ 1180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:23.080065 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:23.080081 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:25:23.080159 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:23.080164 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:23.081189 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:23.081192 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:23.082450 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:23.082460 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:23.083447 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:23.083451 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:23.083531 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:23.083554 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 04:25:23.083624 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 04:25:23.083691 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"mutex_wait_us":26,"thread_start_us":107,"threads_started":1}
I20260620 04:25:23.084197 4961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:23.078713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38260 (local address 127.0.98.2:36043)
0620 04:25:23.079618 (+ 905us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:23.079629 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:23.079652 (+ 23us) server_negotiation.cc:407] Connection header received
0620 04:25:23.079744 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:23.079749 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:23.079920 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:23.080089 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:25:23.080232 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:23.081014 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:23.082571 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:23.083323 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:23.083422 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:23.083555 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:23.083981 (+ 426us) server_negotiation.cc:299] Negotiation successful
0620 04:25:23.084075 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":767,"thread_start_us":106,"threads_started":1}
I20260620 04:25:23.086716 4880 tablet_service.cc:1467] Processing CreateTablet for tablet 035aea09b2a44b91944dbc1a48039daf (DEFAULT_TABLE table=test-table [id=24d094bf63504f4b83aaa981c31f67d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:25:23.086719 4747 tablet_service.cc:1467] Processing CreateTablet for tablet 035aea09b2a44b91944dbc1a48039daf (DEFAULT_TABLE table=test-table [id=24d094bf63504f4b83aaa981c31f67d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:25:23.087610 4747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 035aea09b2a44b91944dbc1a48039daf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:25:23.087617 4880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 035aea09b2a44b91944dbc1a48039daf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:25:23.088335 4959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:23.078489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40965 (local address 127.0.0.1:48834)
0620 04:25:23.079082 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:23.079094 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:23.079162 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:23.079745 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:23.079748 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:23.079765 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:25:23.079861 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:23.079867 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:23.081127 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:23.081131 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:23.082454 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:23.082460 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:23.084612 (+ 2152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:23.084616 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:23.084702 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:23.088013 (+ 3311us) client_negotiation.cc:769] Sending connection context
0620 04:25:23.088100 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 04:25:23.088189 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"mutex_wait_us":63,"thread_start_us":94,"threads_started":1}
I20260620 04:25:23.088408 4962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:23.078861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48834 (local address 127.0.98.1:40965)
0620 04:25:23.079149 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:23.079159 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:23.079180 (+ 21us) server_negotiation.cc:407] Connection header received
0620 04:25:23.079248 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:23.079254 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:23.079434 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:23.079590 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:25:23.080005 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:23.081013 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:23.083617 (+ 2604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:23.084495 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:23.084601 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:23.084726 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:23.088148 (+ 3422us) server_negotiation.cc:299] Negotiation successful
0620 04:25:23.088260 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":112,"threads_started":1}
I20260620 04:25:23.091483 4614 tablet_service.cc:1467] Processing CreateTablet for tablet 035aea09b2a44b91944dbc1a48039daf (DEFAULT_TABLE table=test-table [id=24d094bf63504f4b83aaa981c31f67d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:25:23.092751 4614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 035aea09b2a44b91944dbc1a48039daf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:25:23.094070 4963 tablet_bootstrap.cc:492] T 035aea09b2a44b91944dbc1a48039daf P e79b9c19c2cd453880e41e81b3e0eb9f: Bootstrap starting.
I20260620 04:25:23.095274 4964 tablet_bootstrap.cc:492] T 035aea09b2a44b91944dbc1a48039daf P b36a6e673aeb44338eaefc2db24c5280: Bootstrap starting.
I20260620 04:25:23.096266 4963 tablet_bootstrap.cc:654] T 035aea09b2a44b91944dbc1a48039daf P e79b9c19c2cd453880e41e81b3e0eb9f: Neither blocks nor log segments found. Creating new log.
I20260620 04:25:23.097373 4964 tablet_bootstrap.cc:654] T 035aea09b2a44b91944dbc1a48039daf P b36a6e673aeb44338eaefc2db24c5280: Neither blocks nor log segments found. Creating new log.
I20260620 04:25:23.097568 4963 log.cc:824] T 035aea09b2a44b91944dbc1a48039daf P e79b9c19c2cd453880e41e81b3e0eb9f: Log is configured to *not* fsync() on all Append() calls
I20260620 04:25:23.098502 4964 log.cc:824] T 035aea09b2a44b91944dbc1a48039daf P b36a6e673aeb44338eaefc2db24c5280: Log is configured to *not* fsync() on all Append() calls
I20260620 04:25:23.099427 4963 tablet_bootstrap.cc:492] T 035aea09b2a44b91944dbc1a48039daf P e79b9c19c2cd453880e41e81b3e0eb9f: No bootstrap required, opened a new log
I20260620 04:25:23.099611 4963 ts_tablet_manager.cc:1397] T 035aea09b2a44b91944dbc1a48039daf P e79b9c19c2cd453880e41e81b3e0eb9f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 04:25:23.100455 4964 tablet_bootstrap.cc:492] T 035aea09b2a44b91944dbc1a48039daf P b36a6e673aeb44338eaefc2db24c5280: No bootstrap required, opened a new log
I20260620 04:25:23.100637 4964 ts_tablet_manager.cc:1397] T 035aea09b2a44b91944dbc1a48039daf P b36a6e673aeb44338eaefc2db24c5280: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 04:25:23.100725 4967 tablet_bootstrap.cc:492] T 035aea09b2a44b91944dbc1a48039daf P a2c69b44b06042c39176ed40803f060b: Bootstrap starting.
I20260620 04:25:23.102815 4967 tablet_bootstrap.cc:654] T 035aea09b2a44b91944dbc1a48039daf P a2c69b44b06042c39176ed40803f060b: Neither blocks nor log segments found. Creating new log.
I20260620 04:25:23.103157 4963 raft_consensus.cc:348] T 035aea09b2a44b91944dbc1a48039daf P e79b9c19c2cd453880e41e81b3e0eb9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79b9c19c2cd453880e41e81b3e0eb9f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42851 } } peers { permanent_uuid: "a2c69b44b06042c39176ed40803f060b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40965 } } peers { permanent_uuid: "b36a6e673aeb44338eaefc2db24c5280" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36043 } }
I20260620 04:25:23.103394 4963 raft_consensus.cc:374] T 035aea09b2a44b91944dbc1a48039daf P e79b9c19c2cd453880e41e81b3e0eb9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:25:23.103466 4963 raft_consensus.cc:729] T 035aea09b2a44b91944dbc1a48039daf P e79b9c19c2cd453880e41e81b3e0eb9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e79b9c19c2cd453880e41e81b3e0eb9f, State: Initialized, Role: FOLLOWER
I20260620 04:25:23.103880 4963 consensus_queue.cc:260] T 035aea09b2a44b91944dbc1a48039daf P e79b9c19c2cd453880e41e81b3e0eb9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79b9c19c2cd453880e41e81b3e0eb9f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42851 } } peers { permanent_uuid: "a2c69b44b06042c39176ed40803f060b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40965 } } peers { permanent_uuid: "b36a6e673aeb44338eaefc2db24c5280" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36043 } }
I20260620 04:25:23.104103 4967 log.cc:824] T 035aea09b2a44b91944dbc1a48039daf P a2c69b44b06042c39176ed40803f060b: Log is configured to *not* fsync() on all Append() calls
I20260620 04:25:23.104430 4964 raft_consensus.cc:348] T 035aea09b2a44b91944dbc1a48039daf P b36a6e673aeb44338eaefc2db24c5280 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79b9c19c2cd453880e41e81b3e0eb9f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42851 } } peers { permanent_uuid: "a2c69b44b06042c39176ed40803f060b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40965 } } peers { permanent_uuid: "b36a6e673aeb44338eaefc2db24c5280" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36043 } }
I20260620 04:25:23.104773 4964 raft_consensus.cc:374] T 035aea09b2a44b91944dbc1a48039daf P b36a6e673aeb44338eaefc2db24c5280 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:25:23.104836 4964 raft_consensus.cc:729] T 035aea09b2a44b91944dbc1a48039daf P b36a6e673aeb44338eaefc2db24c5280 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b36a6e673aeb44338eaefc2db24c5280, State: Initialized, Role: FOLLOWER
I20260620 04:25:23.104837 4946 heartbeater.cc:503] Master 127.0.98.62:41517 was elected leader, sending a full tablet report...
I20260620 04:25:23.104863 4963 ts_tablet_manager.cc:1428] T 035aea09b2a44b91944dbc1a48039daf P e79b9c19c2cd453880e41e81b3e0eb9f: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 04:25:23.105281 4964 consensus_queue.cc:260] T 035aea09b2a44b91944dbc1a48039daf P b36a6e673aeb44338eaefc2db24c5280 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79b9c19c2cd453880e41e81b3e0eb9f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42851 } } peers { permanent_uuid: "a2c69b44b06042c39176ed40803f060b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40965 } } peers { permanent_uuid: "b36a6e673aeb44338eaefc2db24c5280" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36043 } }
I20260620 04:25:23.105880 4813 heartbeater.cc:503] Master 127.0.98.62:41517 was elected leader, sending a full tablet report...
I20260620 04:25:23.106242 4964 ts_tablet_manager.cc:1428] T 035aea09b2a44b91944dbc1a48039daf P b36a6e673aeb44338eaefc2db24c5280: Time spent starting tablet: real 0.006s user 0.002s sys 0.003s
I20260620 04:25:23.106277 4967 tablet_bootstrap.cc:492] T 035aea09b2a44b91944dbc1a48039daf P a2c69b44b06042c39176ed40803f060b: No bootstrap required, opened a new log
I20260620 04:25:23.106441 4967 ts_tablet_manager.cc:1397] T 035aea09b2a44b91944dbc1a48039daf P a2c69b44b06042c39176ed40803f060b: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260620 04:25:23.109607 4967 raft_consensus.cc:348] T 035aea09b2a44b91944dbc1a48039daf P a2c69b44b06042c39176ed40803f060b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79b9c19c2cd453880e41e81b3e0eb9f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42851 } } peers { permanent_uuid: "a2c69b44b06042c39176ed40803f060b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40965 } } peers { permanent_uuid: "b36a6e673aeb44338eaefc2db24c5280" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36043 } }
I20260620 04:25:23.109862 4967 raft_consensus.cc:374] T 035aea09b2a44b91944dbc1a48039daf P a2c69b44b06042c39176ed40803f060b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:25:23.109918 4967 raft_consensus.cc:729] T 035aea09b2a44b91944dbc1a48039daf P a2c69b44b06042c39176ed40803f060b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2c69b44b06042c39176ed40803f060b, State: Initialized, Role: FOLLOWER
I20260620 04:25:23.110329 4967 consensus_queue.cc:260] T 035aea09b2a44b91944dbc1a48039daf P a2c69b44b06042c39176ed40803f060b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79b9c19c2cd453880e41e81b3e0eb9f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42851 } } peers { permanent_uuid: "a2c69b44b06042c39176ed40803f060b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40965 } } peers { permanent_uuid: "b36a6e673aeb44338eaefc2db24c5280" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36043 } }
I20260620 04:25:23.110966 4680 heartbeater.cc:503] Master 127.0.98.62:41517 was elected leader, sending a full tablet report...
I20260620 04:25:23.111150 4967 ts_tablet_manager.cc:1428] T 035aea09b2a44b91944dbc1a48039daf P a2c69b44b06042c39176ed40803f060b: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260620 04:25:23.119122 4968 raft_consensus.cc:482] T 035aea09b2a44b91944dbc1a48039daf P e79b9c19c2cd453880e41e81b3e0eb9f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:25:23.119289 4968 raft_consensus.cc:504] T 035aea09b2a44b91944dbc1a48039daf P e79b9c19c2cd453880e41e81b3e0eb9f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79b9c19c2cd453880e41e81b3e0eb9f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42851 } } peers { permanent_uuid: "a2c69b44b06042c39176ed40803f060b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40965 } } peers { permanent_uuid: "b36a6e673aeb44338eaefc2db24c5280" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36043 } }
I20260620 04:25:23.120273 4968 leader_election.cc:288] T 035aea09b2a44b91944dbc1a48039daf P e79b9c19c2cd453880e41e81b3e0eb9f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2c69b44b06042c39176ed40803f060b (127.0.98.1:40965), b36a6e673aeb44338eaefc2db24c5280 (127.0.98.2:36043)
I20260620 04:25:23.123847 4948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:23.120455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40965 (local address 127.0.98.3:56179)
0620 04:25:23.120574 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:23.120589 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:23.120673 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:23.121036 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:23.121039 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:23.121056 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:25:23.121181 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:23.121188 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:23.121903 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:23.121906 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:23.122708 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:23.122717 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:23.123426 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:23.123430 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:23.123576 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:23.123602 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 04:25:23.123671 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 04:25:23.123740 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260620 04:25:23.123883 4962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:23.120710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56179 (local address 127.0.98.1:40965)
0620 04:25:23.120831 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:23.120835 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:23.120848 (+ 13us) server_negotiation.cc:407] Connection header received
0620 04:25:23.120895 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:23.120898 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:23.120945 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:23.121023 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:25:23.121280 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:23.121806 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:23.122838 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:23.123323 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:23.123411 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:23.123460 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:23.123683 (+ 223us) server_negotiation.cc:299] Negotiation successful
0620 04:25:23.123750 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 04:25:23.124099 4972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:23.120504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36043 (local address 127.0.98.3:33621)
0620 04:25:23.120861 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:23.120873 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:23.120965 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:23.121175 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:23.121178 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:23.121190 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:25:23.121289 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:23.121295 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:23.121988 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:23.121990 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:23.122995 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:23.123005 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:23.123693 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:23.123696 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:23.123792 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:23.123815 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 04:25:23.123887 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 04:25:23.123952 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":114,"threads_started":1}
I20260620 04:25:23.124147 4961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:23.120622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33621 (local address 127.0.98.2:36043)
0620 04:25:23.120738 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:23.120742 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:23.120964 (+ 222us) server_negotiation.cc:407] Connection header received
0620 04:25:23.121043 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:23.121045 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:23.121091 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:23.121166 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:25:23.121385 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:23.121893 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:23.123116 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:23.123579 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:23.123678 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:23.123730 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:23.123912 (+ 182us) server_negotiation.cc:299] Negotiation successful
0620 04:25:23.124027 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260620 04:25:23.124281 4681 tablet.cc:2369] T 035aea09b2a44b91944dbc1a48039daf P a2c69b44b06042c39176ed40803f060b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 04:25:23.124567 4634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "035aea09b2a44b91944dbc1a48039daf" candidate_uuid: "e79b9c19c2cd453880e41e81b3e0eb9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c69b44b06042c39176ed40803f060b" is_pre_election: true
I20260620 04:25:23.124759 4767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "035aea09b2a44b91944dbc1a48039daf" candidate_uuid: "e79b9c19c2cd453880e41e81b3e0eb9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b36a6e673aeb44338eaefc2db24c5280" is_pre_election: true
I20260620 04:25:23.124833 4634 raft_consensus.cc:2454] T 035aea09b2a44b91944dbc1a48039daf P a2c69b44b06042c39176ed40803f060b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e79b9c19c2cd453880e41e81b3e0eb9f in term 0.
I20260620 04:25:23.125003 4767 raft_consensus.cc:2454] T 035aea09b2a44b91944dbc1a48039daf P b36a6e673aeb44338eaefc2db24c5280 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e79b9c19c2cd453880e41e81b3e0eb9f in term 0.
I20260620 04:25:23.125289 4833 leader_election.cc:302] T 035aea09b2a44b91944dbc1a48039daf P e79b9c19c2cd453880e41e81b3e0eb9f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2c69b44b06042c39176ed40803f060b, e79b9c19c2cd453880e41e81b3e0eb9f; no voters:
I20260620 04:25:23.125526 4968 raft_consensus.cc:2781] T 035aea09b2a44b91944dbc1a48039daf P e79b9c19c2cd453880e41e81b3e0eb9f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 04:25:23.125603 4968 raft_consensus.cc:482] T 035aea09b2a44b91944dbc1a48039daf P e79b9c19c2cd453880e41e81b3e0eb9f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 04:25:23.125648 4968 raft_consensus.cc:3037] T 035aea09b2a44b91944dbc1a48039daf P e79b9c19c2cd453880e41e81b3e0eb9f [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:25:23.126595 4968 raft_consensus.cc:504] T 035aea09b2a44b91944dbc1a48039daf P e79b9c19c2cd453880e41e81b3e0eb9f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79b9c19c2cd453880e41e81b3e0eb9f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42851 } } peers { permanent_uuid: "a2c69b44b06042c39176ed40803f060b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40965 } } peers { permanent_uuid: "b36a6e673aeb44338eaefc2db24c5280" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36043 } }
I20260620 04:25:23.126904 4968 leader_election.cc:288] T 035aea09b2a44b91944dbc1a48039daf P e79b9c19c2cd453880e41e81b3e0eb9f [CANDIDATE]: Term 1 election: Requested vote from peers a2c69b44b06042c39176ed40803f060b (127.0.98.1:40965), b36a6e673aeb44338eaefc2db24c5280 (127.0.98.2:36043)
I20260620 04:25:23.127228 4767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "035aea09b2a44b91944dbc1a48039daf" candidate_uuid: "e79b9c19c2cd453880e41e81b3e0eb9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b36a6e673aeb44338eaefc2db24c5280"
I20260620 04:25:23.127228 4634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "035aea09b2a44b91944dbc1a48039daf" candidate_uuid: "e79b9c19c2cd453880e41e81b3e0eb9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c69b44b06042c39176ed40803f060b"
I20260620 04:25:23.127331 4767 raft_consensus.cc:3037] T 035aea09b2a44b91944dbc1a48039daf P b36a6e673aeb44338eaefc2db24c5280 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:25:23.127336 4634 raft_consensus.cc:3037] T 035aea09b2a44b91944dbc1a48039daf P a2c69b44b06042c39176ed40803f060b [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:25:23.128510 4767 raft_consensus.cc:2454] T 035aea09b2a44b91944dbc1a48039daf P b36a6e673aeb44338eaefc2db24c5280 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e79b9c19c2cd453880e41e81b3e0eb9f in term 1.
I20260620 04:25:23.128618 4634 raft_consensus.cc:2454] T 035aea09b2a44b91944dbc1a48039daf P a2c69b44b06042c39176ed40803f060b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e79b9c19c2cd453880e41e81b3e0eb9f in term 1.
I20260620 04:25:23.128777 4836 leader_election.cc:302] T 035aea09b2a44b91944dbc1a48039daf P e79b9c19c2cd453880e41e81b3e0eb9f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b36a6e673aeb44338eaefc2db24c5280, e79b9c19c2cd453880e41e81b3e0eb9f; no voters:
I20260620 04:25:23.128940 4968 raft_consensus.cc:2781] T 035aea09b2a44b91944dbc1a48039daf P e79b9c19c2cd453880e41e81b3e0eb9f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:25:23.129138 4968 raft_consensus.cc:686] T 035aea09b2a44b91944dbc1a48039daf P e79b9c19c2cd453880e41e81b3e0eb9f [term 1 LEADER]: Becoming Leader. State: Replica: e79b9c19c2cd453880e41e81b3e0eb9f, State: Running, Role: LEADER
I20260620 04:25:23.129426 4968 consensus_queue.cc:237] T 035aea09b2a44b91944dbc1a48039daf P e79b9c19c2cd453880e41e81b3e0eb9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79b9c19c2cd453880e41e81b3e0eb9f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42851 } } peers { permanent_uuid: "a2c69b44b06042c39176ed40803f060b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40965 } } peers { permanent_uuid: "b36a6e673aeb44338eaefc2db24c5280" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36043 } }
I20260620 04:25:23.132558 4488 catalog_manager.cc:5495] T 035aea09b2a44b91944dbc1a48039daf P e79b9c19c2cd453880e41e81b3e0eb9f reported cstate change: term changed from 0 to 1, leader changed from <none> to e79b9c19c2cd453880e41e81b3e0eb9f (127.0.98.3). New cstate: current_term: 1 leader_uuid: "e79b9c19c2cd453880e41e81b3e0eb9f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79b9c19c2cd453880e41e81b3e0eb9f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42851 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2c69b44b06042c39176ed40803f060b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b36a6e673aeb44338eaefc2db24c5280" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36043 } health_report { overall_health: UNKNOWN } } }
I20260620 04:25:23.148916 4958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:23.145644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53940 (local address 127.0.98.3:42851)
0620 04:25:23.145752 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:23.145756 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:23.145768 (+ 12us) server_negotiation.cc:407] Connection header received
0620 04:25:23.145884 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:23.145886 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:23.145938 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:23.146022 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:25:23.146221 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:23.146716 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:23.147516 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:23.147736 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:23.147814 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:23.148127 (+ 313us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:25:23.148558 (+ 431us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:25:23.148649 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:23.148710 (+ 61us) server_negotiation.cc:299] Negotiation successful
0620 04:25:23.148758 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 04:25:23.154811 4767 raft_consensus.cc:1261] T 035aea09b2a44b91944dbc1a48039daf P b36a6e673aeb44338eaefc2db24c5280 [term 1 FOLLOWER]: Refusing update from remote peer e79b9c19c2cd453880e41e81b3e0eb9f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:25:23.154843 4634 raft_consensus.cc:1261] T 035aea09b2a44b91944dbc1a48039daf P a2c69b44b06042c39176ed40803f060b [term 1 FOLLOWER]: Refusing update from remote peer e79b9c19c2cd453880e41e81b3e0eb9f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:25:23.155495 4968 consensus_queue.cc:1035] T 035aea09b2a44b91944dbc1a48039daf P e79b9c19c2cd453880e41e81b3e0eb9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b36a6e673aeb44338eaefc2db24c5280" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:25:23.155678 4973 consensus_queue.cc:1035] T 035aea09b2a44b91944dbc1a48039daf P e79b9c19c2cd453880e41e81b3e0eb9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2c69b44b06042c39176ed40803f060b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:25:23.163260 4978 mvcc.cc:205] Tried to move back new op lower bound from 7298783326834692096 to 7298783326740500480. Current Snapshot: MvccSnapshot[applied={T|T < 7298783326834692096}]
I20260620 04:25:23.165450 4977 mvcc.cc:205] Tried to move back new op lower bound from 7298783326834692096 to 7298783326740500480. Current Snapshot: MvccSnapshot[applied={T|T < 7298783326834692096 or (T in {7298783326834692096})}]
I20260620 04:25:23.171124 4489 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35264:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:25:23.171267 4489 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35264:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:25:23.173449 4489 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fdf38559144d41bfb61537a0db395043: Sending DeleteTablet for 3 replicas of tablet 035aea09b2a44b91944dbc1a48039daf
I20260620 04:25:23.174198 4614 tablet_service.cc:1514] Processing DeleteTablet for tablet 035aea09b2a44b91944dbc1a48039daf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:25:23 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48834
I20260620 04:25:23.174350 4747 tablet_service.cc:1514] Processing DeleteTablet for tablet 035aea09b2a44b91944dbc1a48039daf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:25:23 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38260
I20260620 04:25:23.174795 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 4552
I20260620 04:25:23.174863 4987 tablet_replica.cc:330] T 035aea09b2a44b91944dbc1a48039daf P b36a6e673aeb44338eaefc2db24c5280: stopping tablet replica
I20260620 04:25:23.174868 4880 tablet_service.cc:1514] Processing DeleteTablet for tablet 035aea09b2a44b91944dbc1a48039daf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:25:23 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53934
I20260620 04:25:23.175175 4987 raft_consensus.cc:2229] T 035aea09b2a44b91944dbc1a48039daf P b36a6e673aeb44338eaefc2db24c5280 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:25:23.175489 4987 raft_consensus.cc:2258] T 035aea09b2a44b91944dbc1a48039daf P b36a6e673aeb44338eaefc2db24c5280 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:25:23.177057 4987 ts_tablet_manager.cc:1905] T 035aea09b2a44b91944dbc1a48039daf P b36a6e673aeb44338eaefc2db24c5280: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:25:23.177079 4988 tablet_replica.cc:330] T 035aea09b2a44b91944dbc1a48039daf P e79b9c19c2cd453880e41e81b3e0eb9f: stopping tablet replica
I20260620 04:25:23.177318 4988 raft_consensus.cc:2229] T 035aea09b2a44b91944dbc1a48039daf P e79b9c19c2cd453880e41e81b3e0eb9f [term 1 LEADER]: Raft consensus shutting down.
I20260620 04:25:23.177678 4988 raft_consensus.cc:2258] T 035aea09b2a44b91944dbc1a48039daf P e79b9c19c2cd453880e41e81b3e0eb9f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:25:23.179606 4988 ts_tablet_manager.cc:1905] T 035aea09b2a44b91944dbc1a48039daf P e79b9c19c2cd453880e41e81b3e0eb9f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:25:23.179675 4987 ts_tablet_manager.cc:1918] T 035aea09b2a44b91944dbc1a48039daf P b36a6e673aeb44338eaefc2db24c5280: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:25:23.179747 4987 log.cc:1196] T 035aea09b2a44b91944dbc1a48039daf P b36a6e673aeb44338eaefc2db24c5280: Deleting WAL directory at /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929496330398-392-0/minicluster-data/ts-1/wal/wals/035aea09b2a44b91944dbc1a48039daf
I20260620 04:25:23.180083 4987 ts_tablet_manager.cc:1939] T 035aea09b2a44b91944dbc1a48039daf P b36a6e673aeb44338eaefc2db24c5280: Deleting consensus metadata
I20260620 04:25:23.181257 4477 catalog_manager.cc:4841] TS b36a6e673aeb44338eaefc2db24c5280 (127.0.98.2:36043): tablet 035aea09b2a44b91944dbc1a48039daf (table test-table [id=24d094bf63504f4b83aaa981c31f67d3]) successfully deleted
I20260620 04:25:23.181938 4988 ts_tablet_manager.cc:1918] T 035aea09b2a44b91944dbc1a48039daf P e79b9c19c2cd453880e41e81b3e0eb9f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:25:23.182017 4988 log.cc:1196] T 035aea09b2a44b91944dbc1a48039daf P e79b9c19c2cd453880e41e81b3e0eb9f: Deleting WAL directory at /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929496330398-392-0/minicluster-data/ts-2/wal/wals/035aea09b2a44b91944dbc1a48039daf
I20260620 04:25:23.182260 4988 ts_tablet_manager.cc:1939] T 035aea09b2a44b91944dbc1a48039daf P e79b9c19c2cd453880e41e81b3e0eb9f: Deleting consensus metadata
W20260620 04:25:23.183145 4474 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40965 (error 108)
I20260620 04:25:23.183590 4474 catalog_manager.cc:4841] TS e79b9c19c2cd453880e41e81b3e0eb9f (127.0.98.3:42851): tablet 035aea09b2a44b91944dbc1a48039daf (table test-table [id=24d094bf63504f4b83aaa981c31f67d3]) successfully deleted
I20260620 04:25:23.184114 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 4685
W20260620 04:25:23.184295 4959 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:25:23.183859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40965 (local address 127.0.0.1:48846)
0620 04:25:23.184036 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:23.184050 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:23.184136 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:23.184210 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40965: BlockingWrite error: write error: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":107}
W20260620 04:25:23.184739 4474 catalog_manager.cc:4584] TS a2c69b44b06042c39176ed40803f060b (127.0.98.1:40965): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 035aea09b2a44b91944dbc1a48039daf: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40965: BlockingWrite error: write error: Connection reset by peer (error 104)
I20260620 04:25:23.189317 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 4818
I20260620 04:25:23.193955 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 4459
2026-06-20T04:25:23Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2061 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-20T04:25:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:25:23Z Disabled control of system clock
I20260620 04:25:23.221390 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929496330398-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929496330398-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929496330398-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44867
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:45919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44867
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 04:25:23.329423 4993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:25:23.329710 4993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:25:23.329770 4993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:25:23.333778 4993 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:25:23.333856 4993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:25:23.333882 4993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:25:23.333909 4993 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:25:23.333943 4993 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:25:23.338881 4993 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929496330398-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929496330398-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44867
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44867
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4993
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929496330398-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:25:23.340022 4993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:25:23.341377 4993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 04:25:23.349303 4993 server_base.cc:1034] running on GCE node
W20260620 04:25:23.349308 4999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:25:23.349260 4998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:25:23.349260 5001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:25:23.350036 4993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:25:23.351056 4993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:25:23.352234 4993 hybrid_clock.cc:648] HybridClock initialized: now 1781929523352213 us; error 27 us; skew 500 ppm
I20260620 04:25:23.354084 4993 webserver.cc:460] Webserver started at http://127.0.98.62:34339/ using document root <none> and password file <none>
I20260620 04:25:23.354611 4993 fs_manager.cc:359] Metadata directory not provided
I20260620 04:25:23.354672 4993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:25:23.354849 4993 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:25:23.356900 4993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929496330398-392-0/minicluster-data/master-0/data/instance:
uuid: "59aaa1415d1a45d58583d019b6a650a1"
format_stamp: "Formatted at 2026-06-20 04:25:23 on dist-test-slave-nk6d"
server_key: "9850191d21db282acb3d17f541797984"
server_key_iv: "17ff874cc61e429def64d3a99b41d0ab"
server_key_version: "encryptionkey@0"
I20260620 04:25:23.357425 4993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929496330398-392-0/minicluster-data/master-0/wal/instance:
uuid: "59aaa1415d1a45d58583d019b6a650a1"
format_stamp: "Formatted at 2026-06-20 04:25:23 on dist-test-slave-nk6d"
server_key: "9850191d21db282acb3d17f541797984"
server_key_iv: "17ff874cc61e429def64d3a99b41d0ab"
server_key_version: "encryptionkey@0"
I20260620 04:25:23.361052 4993 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260620 04:25:23.363323 5007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:23.364517 4993 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:25:23.364619 4993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929496330398-392-0/minicluster-data/master-0/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929496330398-392-0/minicluster-data/master-0/wal
uuid: "59aaa1415d1a45d58583d019b6a650a1"
format_stamp: "Formatted at 2026-06-20 04:25:23 on dist-test-slave-nk6d"
server_key: "9850191d21db282acb3d17f541797984"
server_key_iv: "17ff874cc61e429def64d3a99b41d0ab"
server_key_version: "encryptionkey@0"
I20260620 04:25:23.364725 4993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929496330398-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929496330398-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929496330398-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:25:23.369064 4993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:25:23.369621 4993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:25:23.369769 4993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:25:23.376003 4993 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44867
I20260620 04:25:23.375988 5059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44867 every 8 connection(s)
I20260620 04:25:23.376902 4993 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:25:23.379844 5060 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:25:23.384975 5060 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59aaa1415d1a45d58583d019b6a650a1: Bootstrap starting.
I20260620 04:25:23.386682 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 4993
I20260620 04:25:23.386788 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929496330398-392-0/minicluster-data/master-0/wal/instance
I20260620 04:25:23.387032 392 external_mini_cluster.cc:1428] Setting key b27a33370bf10200e1173ddf6b5353ae
I20260620 04:25:23.387326 5060 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 59aaa1415d1a45d58583d019b6a650a1: Neither blocks nor log segments found. Creating new log.
I20260620 04:25:23.388186 5060 log.cc:824] T 00000000000000000000000000000000 P 59aaa1415d1a45d58583d019b6a650a1: Log is configured to *not* fsync() on all Append() calls
I20260620 04:25:23.390509 5060 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59aaa1415d1a45d58583d019b6a650a1: No bootstrap required, opened a new log
I20260620 04:25:23.392894 5063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:23.388092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47024 (local address 127.0.98.62:44867)
0620 04:25:23.388772 (+ 680us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:23.388787 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:23.388823 (+ 36us) server_negotiation.cc:407] Connection header received
0620 04:25:23.389552 (+ 729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:23.389588 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:23.389943 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:23.390303 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:23.391324 (+ 1021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:23.391329 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:23.391345 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:25:23.391356 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:23.391426 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:25:23.391449 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:23.391570 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:23.391677 (+ 107us) server_negotiation.cc:299] Negotiation successful
0620 04:25:23.391922 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":119,"threads_started":1}
I20260620 04:25:23.394809 5060 raft_consensus.cc:348] T 00000000000000000000000000000000 P 59aaa1415d1a45d58583d019b6a650a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59aaa1415d1a45d58583d019b6a650a1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44867 } }
I20260620 04:25:23.395077 5060 raft_consensus.cc:374] T 00000000000000000000000000000000 P 59aaa1415d1a45d58583d019b6a650a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:25:23.395133 5060 raft_consensus.cc:729] T 00000000000000000000000000000000 P 59aaa1415d1a45d58583d019b6a650a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59aaa1415d1a45d58583d019b6a650a1, State: Initialized, Role: FOLLOWER
I20260620 04:25:23.395546 5060 consensus_queue.cc:260] T 00000000000000000000000000000000 P 59aaa1415d1a45d58583d019b6a650a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59aaa1415d1a45d58583d019b6a650a1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44867 } }
I20260620 04:25:23.395669 5060 raft_consensus.cc:388] T 00000000000000000000000000000000 P 59aaa1415d1a45d58583d019b6a650a1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:25:23.395727 5060 raft_consensus.cc:482] T 00000000000000000000000000000000 P 59aaa1415d1a45d58583d019b6a650a1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:25:23.395815 5060 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 59aaa1415d1a45d58583d019b6a650a1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:25:23.396732 5060 raft_consensus.cc:504] T 00000000000000000000000000000000 P 59aaa1415d1a45d58583d019b6a650a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59aaa1415d1a45d58583d019b6a650a1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44867 } }
I20260620 04:25:23.397058 5060 leader_election.cc:302] T 00000000000000000000000000000000 P 59aaa1415d1a45d58583d019b6a650a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 59aaa1415d1a45d58583d019b6a650a1; no voters:
I20260620 04:25:23.397329 5060 leader_election.cc:288] T 00000000000000000000000000000000 P 59aaa1415d1a45d58583d019b6a650a1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:25:23.397462 5065 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 59aaa1415d1a45d58583d019b6a650a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:25:23.397679 5065 raft_consensus.cc:686] T 00000000000000000000000000000000 P 59aaa1415d1a45d58583d019b6a650a1 [term 1 LEADER]: Becoming Leader. State: Replica: 59aaa1415d1a45d58583d019b6a650a1, State: Running, Role: LEADER
I20260620 04:25:23.397997 5065 consensus_queue.cc:237] T 00000000000000000000000000000000 P 59aaa1415d1a45d58583d019b6a650a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59aaa1415d1a45d58583d019b6a650a1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44867 } }
I20260620 04:25:23.398551 5060 sys_catalog.cc:564] T 00000000000000000000000000000000 P 59aaa1415d1a45d58583d019b6a650a1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:25:23.399076 5065 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59aaa1415d1a45d58583d019b6a650a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 59aaa1415d1a45d58583d019b6a650a1. Latest consensus state: current_term: 1 leader_uuid: "59aaa1415d1a45d58583d019b6a650a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59aaa1415d1a45d58583d019b6a650a1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44867 } } }
I20260620 04:25:23.399186 5065 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59aaa1415d1a45d58583d019b6a650a1 [sys.catalog]: This master's current role is: LEADER
I20260620 04:25:23.399770 5066 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59aaa1415d1a45d58583d019b6a650a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "59aaa1415d1a45d58583d019b6a650a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59aaa1415d1a45d58583d019b6a650a1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44867 } } }
I20260620 04:25:23.399899 5066 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59aaa1415d1a45d58583d019b6a650a1 [sys.catalog]: This master's current role is: LEADER
I20260620 04:25:23.400002 5071 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:25:23.403574 5071 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:25:23.408816 5071 catalog_manager.cc:1347] Generated new cluster ID: 4f93c31aa89d49a0ab34a5e74aea78d7
I20260620 04:25:23.408872 5071 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:25:23.421207 5071 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:25:23.421533 5071 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:25:23.427626 5071 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 59aaa1415d1a45d58583d019b6a650a1: Generated new TSK 0
I20260620 04:25:23.428383 5071 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 04:25:23.434630 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929496330398-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929496330398-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929496330398-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929496330398-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44867
--builtin_ntp_servers=127.0.98.20:45919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 04:25:23.542268 5084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:25:23.542608 5084 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:25:23.542678 5084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:25:23.548295 5084 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:25:23.548367 5084 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:25:23.548450 5084 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 04:25:23.553558 5084 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929496330398-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929496330398-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929496330398-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44867
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5084
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929496330398-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:25:23.554593 5084 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:25:23.555876 5084 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:25:23.564849 5092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:25:23.564858 5089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:25:23.564870 5090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:25:23.565095 5084 server_base.cc:1034] running on GCE node
I20260620 04:25:23.565510 5084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:25:23.566098 5084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:25:23.567283 5084 hybrid_clock.cc:648] HybridClock initialized: now 1781929523567253 us; error 42 us; skew 500 ppm
I20260620 04:25:23.568944 5084 webserver.cc:460] Webserver started at http://127.0.98.1:45219/ using document root <none> and password file <none>
I20260620 04:25:23.569463 5084 fs_manager.cc:359] Metadata directory not provided
I20260620 04:25:23.569525 5084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:25:23.569695 5084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:25:23.571623 5084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929496330398-392-0/minicluster-data/ts-0/data/instance:
uuid: "b311d398ae3242789dc406fcfa938cc6"
format_stamp: "Formatted at 2026-06-20 04:25:23 on dist-test-slave-nk6d"
server_key: "fbf280fe4866cf7022cb33d7ce710a5c"
server_key_iv: "e8f9265bb8420de159266e153f63c3f2"
server_key_version: "encryptionkey@0"
I20260620 04:25:23.572167 5084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929496330398-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b311d398ae3242789dc406fcfa938cc6"
format_stamp: "Formatted at 2026-06-20 04:25:23 on dist-test-slave-nk6d"
server_key: "fbf280fe4866cf7022cb33d7ce710a5c"
server_key_iv: "e8f9265bb8420de159266e153f63c3f2"
server_key_version: "encryptionkey@0"
I20260620 04:25:23.575536 5084 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260620 04:25:23.577850 5098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:23.579118 5084 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 04:25:23.579236 5084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929496330398-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929496330398-392-0/minicluster-data/ts-0/wal
uuid: "b311d398ae3242789dc406fcfa938cc6"
format_stamp: "Formatted at 2026-06-20 04:25:23 on dist-test-slave-nk6d"
server_key: "fbf280fe4866cf7022cb33d7ce710a5c"
server_key_iv: "e8f9265bb8420de159266e153f63c3f2"
server_key_version: "encryptionkey@0"
I20260620 04:25:23.579344 5084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929496330398-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929496330398-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929496330398-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:25:23.583772 5084 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:25:23.584465 5084 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:25:23.584645 5084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:25:23.585150 5084 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:25:23.586025 5084 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:25:23.586069 5084 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:23.586118 5084 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:25:23.586160 5084 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:23.594375 5084 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39687
I20260620 04:25:23.594395 5211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39687 every 8 connection(s)
I20260620 04:25:23.595237 5084 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929496330398-392-0/minicluster-data/ts-0/data/info.pb
I20260620 04:25:23.600359 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 5084
I20260620 04:25:23.600476 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929496330398-392-0/minicluster-data/ts-0/wal/instance
I20260620 04:25:23.600731 392 external_mini_cluster.cc:1428] Setting key d1d8aad4624ce55a08e119fde45b2076
I20260620 04:25:23.601596 5063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:23.597030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34783 (local address 127.0.98.62:44867)
0620 04:25:23.597161 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:23.597164 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:23.598392 (+ 1228us) server_negotiation.cc:407] Connection header received
0620 04:25:23.599752 (+ 1360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:23.599754 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:23.599809 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:23.599879 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:23.601128 (+ 1249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:23.601132 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:23.601134 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:25:23.601147 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:23.601171 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:25:23.601181 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:23.601265 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:23.601433 (+ 168us) server_negotiation.cc:299] Negotiation successful
0620 04:25:23.601474 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 04:25:23.602473 5214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:23.597549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44867 (local address 127.0.98.1:34783)
0620 04:25:23.598240 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:23.598281 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:23.599543 (+ 1262us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:23.600105 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:23.600112 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:23.600219 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:23.600816 (+ 597us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 04:25:23.600818 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:23.600869 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 04:25:23.600984 (+ 115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:25:23.601266 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:25:23.601275 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:25:23.601281 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 04:25:23.601467 (+ 186us) client_negotiation.cc:240] Negotiation successful
0620 04:25:23.601666 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":108,"threads_started":1}
I20260620 04:25:23.602564 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929496330398-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929496330398-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929496330398-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929496330398-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44867
--builtin_ntp_servers=127.0.98.20:45919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260620 04:25:23.604566 5212 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44867
I20260620 04:25:23.604796 5212 heartbeater.cc:464] Registering TS with master...
I20260620 04:25:23.605160 5212 heartbeater.cc:511] Master 127.0.98.62:44867 requested a full tablet report, sending...
I20260620 04:25:23.606756 5024 ts_manager.cc:194] Registered new tserver with Master: b311d398ae3242789dc406fcfa938cc6 (127.0.98.1:39687)
W20260620 04:25:23.706835 5215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:25:23.707084 5215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:25:23.707121 5215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:25:23.711000 5215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:25:23.711051 5215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:25:23.711104 5215 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 04:25:23.715644 5215 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929496330398-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929496330398-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929496330398-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44867
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5215
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929496330398-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:25:23.716609 5215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:25:23.717880 5215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:25:23.726781 5221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:25:23.726826 5223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:25:23.726788 5220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:25:23.727288 5215 server_base.cc:1034] running on GCE node
I20260620 04:25:23.727654 5215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:25:23.728289 5215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:25:23.729435 5215 hybrid_clock.cc:648] HybridClock initialized: now 1781929523729432 us; error 46 us; skew 500 ppm
I20260620 04:25:23.731153 5215 webserver.cc:460] Webserver started at http://127.0.98.2:45847/ using document root <none> and password file <none>
I20260620 04:25:23.731693 5215 fs_manager.cc:359] Metadata directory not provided
I20260620 04:25:23.731751 5215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:25:23.731928 5215 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:25:23.733932 5215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929496330398-392-0/minicluster-data/ts-1/data/instance:
uuid: "dfe9bbabeffd4361af69b770a5b530ac"
format_stamp: "Formatted at 2026-06-20 04:25:23 on dist-test-slave-nk6d"
server_key: "a7be90de96eccc1c1dcd179e1dea2bfc"
server_key_iv: "2afa4d0760dd45caa70d1ba849215e6b"
server_key_version: "encryptionkey@0"
I20260620 04:25:23.734468 5215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929496330398-392-0/minicluster-data/ts-1/wal/instance:
uuid: "dfe9bbabeffd4361af69b770a5b530ac"
format_stamp: "Formatted at 2026-06-20 04:25:23 on dist-test-slave-nk6d"
server_key: "a7be90de96eccc1c1dcd179e1dea2bfc"
server_key_iv: "2afa4d0760dd45caa70d1ba849215e6b"
server_key_version: "encryptionkey@0"
I20260620 04:25:23.737985 5215 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260620 04:25:23.740240 5229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:23.741547 5215 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 04:25:23.741671 5215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929496330398-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929496330398-392-0/minicluster-data/ts-1/wal
uuid: "dfe9bbabeffd4361af69b770a5b530ac"
format_stamp: "Formatted at 2026-06-20 04:25:23 on dist-test-slave-nk6d"
server_key: "a7be90de96eccc1c1dcd179e1dea2bfc"
server_key_iv: "2afa4d0760dd45caa70d1ba849215e6b"
server_key_version: "encryptionkey@0"
I20260620 04:25:23.741772 5215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929496330398-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929496330398-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929496330398-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:25:23.746419 5215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:25:23.747020 5215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:25:23.747181 5215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:25:23.747727 5215 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:25:23.748651 5215 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:25:23.748701 5215 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:23.748754 5215 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:25:23.748785 5215 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:23.756597 5215 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34415
I20260620 04:25:23.756620 5342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34415 every 8 connection(s)
I20260620 04:25:23.757395 5215 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929496330398-392-0/minicluster-data/ts-1/data/info.pb
I20260620 04:25:23.757714 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 5215
I20260620 04:25:23.757840 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929496330398-392-0/minicluster-data/ts-1/wal/instance
I20260620 04:25:23.758131 392 external_mini_cluster.cc:1428] Setting key 8d94baf4bcc6e63637e73db437c001d6
I20260620 04:25:23.760315 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929496330398-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929496330398-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929496330398-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929496330398-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44867
--builtin_ntp_servers=127.0.98.20:45919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260620 04:25:23.763448 5063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:23.759071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37305 (local address 127.0.98.62:44867)
0620 04:25:23.759192 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:23.759195 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:23.760465 (+ 1270us) server_negotiation.cc:407] Connection header received
0620 04:25:23.761828 (+ 1363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:23.761831 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:23.761870 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:23.761941 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:23.763053 (+ 1112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:23.763056 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:23.763058 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:25:23.763068 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:23.763087 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:25:23.763095 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:23.763183 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:23.763315 (+ 132us) server_negotiation.cc:299] Negotiation successful
0620 04:25:23.763348 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 04:25:23.764561 5345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:23.759628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44867 (local address 127.0.98.2:37305)
0620 04:25:23.760328 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:23.760376 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:23.761625 (+ 1249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:23.762113 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:23.762120 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:23.762206 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:23.762760 (+ 554us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 04:25:23.762762 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:23.762811 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 04:25:23.762891 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:25:23.763182 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:25:23.763189 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:25:23.763195 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 04:25:23.763359 (+ 164us) client_negotiation.cc:240] Negotiation successful
0620 04:25:23.763595 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":101,"threads_started":1}
I20260620 04:25:23.765588 5343 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44867
I20260620 04:25:23.765820 5343 heartbeater.cc:464] Registering TS with master...
I20260620 04:25:23.766294 5343 heartbeater.cc:511] Master 127.0.98.62:44867 requested a full tablet report, sending...
I20260620 04:25:23.767382 5024 ts_manager.cc:194] Registered new tserver with Master: dfe9bbabeffd4361af69b770a5b530ac (127.0.98.2:34415)
W20260620 04:25:23.868578 5346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:25:23.868830 5346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:25:23.868867 5346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:25:23.872745 5346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:25:23.872790 5346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:25:23.872843 5346 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 04:25:23.877373 5346 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929496330398-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929496330398-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929496330398-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44867
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5346
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929496330398-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:25:23.878304 5346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:25:23.879571 5346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:25:23.888594 5352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:25:23.888620 5354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:25:23.888763 5346 server_base.cc:1034] running on GCE node
W20260620 04:25:23.888594 5351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:25:23.889281 5346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:25:23.889882 5346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:25:23.891052 5346 hybrid_clock.cc:648] HybridClock initialized: now 1781929523891024 us; error 36 us; skew 500 ppm
I20260620 04:25:23.892800 5346 webserver.cc:460] Webserver started at http://127.0.98.3:42375/ using document root <none> and password file <none>
I20260620 04:25:23.893321 5346 fs_manager.cc:359] Metadata directory not provided
I20260620 04:25:23.893379 5346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:25:23.893574 5346 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:25:23.895561 5346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929496330398-392-0/minicluster-data/ts-2/data/instance:
uuid: "738a393e040f4ce2bdd6c33fc5045a8a"
format_stamp: "Formatted at 2026-06-20 04:25:23 on dist-test-slave-nk6d"
server_key: "08317c49af0865113a24085c347eddcd"
server_key_iv: "16e783a844fd4c6b50de83233105b561"
server_key_version: "encryptionkey@0"
I20260620 04:25:23.896108 5346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929496330398-392-0/minicluster-data/ts-2/wal/instance:
uuid: "738a393e040f4ce2bdd6c33fc5045a8a"
format_stamp: "Formatted at 2026-06-20 04:25:23 on dist-test-slave-nk6d"
server_key: "08317c49af0865113a24085c347eddcd"
server_key_iv: "16e783a844fd4c6b50de83233105b561"
server_key_version: "encryptionkey@0"
I20260620 04:25:23.899628 5346 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:25:23.902077 5360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:23.903255 5346 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:25:23.903357 5346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929496330398-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929496330398-392-0/minicluster-data/ts-2/wal
uuid: "738a393e040f4ce2bdd6c33fc5045a8a"
format_stamp: "Formatted at 2026-06-20 04:25:23 on dist-test-slave-nk6d"
server_key: "08317c49af0865113a24085c347eddcd"
server_key_iv: "16e783a844fd4c6b50de83233105b561"
server_key_version: "encryptionkey@0"
I20260620 04:25:23.903455 5346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929496330398-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929496330398-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929496330398-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:25:23.907876 5346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:25:23.908447 5346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:25:23.908604 5346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:25:23.909104 5346 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:25:23.909934 5346 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:25:23.909979 5346 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:23.910029 5346 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:25:23.910058 5346 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:23.918052 5346 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38441
I20260620 04:25:23.918087 5473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38441 every 8 connection(s)
I20260620 04:25:23.918908 5346 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929496330398-392-0/minicluster-data/ts-2/data/info.pb
I20260620 04:25:23.924724 5063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:23.920602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44591 (local address 127.0.98.62:44867)
0620 04:25:23.920728 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:23.920732 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:23.921855 (+ 1123us) server_negotiation.cc:407] Connection header received
0620 04:25:23.923111 (+ 1256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:23.923114 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:23.923159 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:23.923234 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:23.924297 (+ 1063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:23.924301 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:23.924304 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:25:23.924316 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:23.924336 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:25:23.924344 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:23.924420 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:23.924540 (+ 120us) server_negotiation.cc:299] Negotiation successful
0620 04:25:23.924583 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 04:25:23.925525 5476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:23.921054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44867 (local address 127.0.98.3:44591)
0620 04:25:23.921701 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:23.921740 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:23.922906 (+ 1166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:23.923416 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:23.923423 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:23.923513 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:23.924042 (+ 529us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 04:25:23.924044 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:23.924085 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 04:25:23.924162 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:25:23.924409 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:25:23.924416 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:25:23.924422 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 04:25:23.924578 (+ 156us) client_negotiation.cc:240] Negotiation successful
0620 04:25:23.924813 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":105,"threads_started":1}
I20260620 04:25:23.925868 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 5346
I20260620 04:25:23.925971 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929496330398-392-0/minicluster-data/ts-2/wal/instance
I20260620 04:25:23.926256 392 external_mini_cluster.cc:1428] Setting key 221b566385224f3b100e22761e54f7e7
I20260620 04:25:23.926709 5474 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44867
I20260620 04:25:23.926934 5474 heartbeater.cc:464] Registering TS with master...
I20260620 04:25:23.927469 5474 heartbeater.cc:511] Master 127.0.98.62:44867 requested a full tablet report, sending...
I20260620 04:25:23.928427 5024 ts_manager.cc:194] Registered new tserver with Master: 738a393e040f4ce2bdd6c33fc5045a8a (127.0.98.3:38441)
I20260620 04:25:23.939760 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 04:25:23.943017 5063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:23.941524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47038 (local address 127.0.98.62:44867)
0620 04:25:23.941697 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:23.941701 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:23.941819 (+ 118us) server_negotiation.cc:407] Connection header received
0620 04:25:23.941933 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:23.941936 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:23.941971 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:23.942031 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:23.942675 (+ 644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:23.942679 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:23.942681 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:25:23.942693 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:23.942713 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:25:23.942722 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:23.942788 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:23.942858 (+ 70us) server_negotiation.cc:299] Negotiation successful
0620 04:25:23.942897 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260620 04:25:23.945473 5024 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47038:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 04:25:23.948549 5024 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 04:25:23.959265 5486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:23.956336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36840 (local address 127.0.98.3:38441)
0620 04:25:23.957241 (+ 905us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:23.957253 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:23.957275 (+ 22us) server_negotiation.cc:407] Connection header received
0620 04:25:23.957324 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:23.957329 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:23.957506 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:23.957643 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:23.958505 (+ 862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:23.958511 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:23.958523 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:25:23.958535 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:23.958578 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:25:23.958615 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:23.958711 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:23.959025 (+ 314us) server_negotiation.cc:299] Negotiation successful
0620 04:25:23.959098 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":808,"thread_start_us":147,"threads_started":1}
I20260620 04:25:23.959376 5489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:23.956879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39687 (local address 127.0.0.1:45348)
0620 04:25:23.957101 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:23.957114 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:23.957174 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:23.958093 (+ 919us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:23.958095 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:23.958109 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:23.958530 (+ 421us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 04:25:23.958531 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:23.958540 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 04:25:23.958558 (+ 18us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:25:23.959106 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:25:23.959109 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:25:23.959113 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 04:25:23.959191 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 04:25:23.959238 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":87,"threads_started":1}
I20260620 04:25:23.959553 5485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:23.956195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38441 (local address 127.0.0.1:36840)
0620 04:25:23.956906 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:23.956920 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:23.957011 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:23.957653 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:23.957659 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:23.957802 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:23.958308 (+ 506us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 04:25:23.958310 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:23.958328 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 04:25:23.958394 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:25:23.958877 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:25:23.958885 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:25:23.958891 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 04:25:23.959023 (+ 132us) client_negotiation.cc:240] Negotiation successful
0620 04:25:23.959235 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"spinlock_wait_cycles":7040,"thread_start_us":66,"threads_started":1}
I20260620 04:25:23.959775 5490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:23.957244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45348 (local address 127.0.98.1:39687)
0620 04:25:23.957734 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:23.957745 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:23.957765 (+ 20us) server_negotiation.cc:407] Connection header received
0620 04:25:23.957811 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:23.957816 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:23.957989 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:23.958110 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:23.958698 (+ 588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:23.958712 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:23.958728 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:25:23.958739 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:23.958774 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:25:23.958802 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:23.958901 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:23.959543 (+ 642us) server_negotiation.cc:299] Negotiation successful
0620 04:25:23.959616 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":102,"threads_started":1}
I20260620 04:25:23.959890 5487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:23.956473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34415 (local address 127.0.0.1:51738)
0620 04:25:23.956872 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:23.956916 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:23.957010 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:23.958695 (+ 1685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:23.958697 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:23.958710 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:23.959091 (+ 381us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 04:25:23.959092 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:23.959101 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 04:25:23.959123 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:25:23.959687 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:25:23.959690 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:25:23.959694 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 04:25:23.959762 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 04:25:23.959793 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"spinlock_wait_cycles":4992,"thread_start_us":113,"threads_started":1}
I20260620 04:25:23.960240 5488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:23.956623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51738 (local address 127.0.98.2:34415)
0620 04:25:23.958330 (+ 1707us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:23.958341 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:23.958362 (+ 21us) server_negotiation.cc:407] Connection header received
0620 04:25:23.958410 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:23.958415 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:23.958586 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:23.958713 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:23.959233 (+ 520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:23.959238 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:23.959247 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:25:23.959256 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:23.959289 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:25:23.959394 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:23.959568 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:23.960056 (+ 488us) server_negotiation.cc:299] Negotiation successful
0620 04:25:23.960123 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1609,"thread_start_us":95,"threads_started":1}
I20260620 04:25:23.962615 5146 tablet_service.cc:1467] Processing CreateTablet for tablet ffc93cd781b84dab89479648d921b944 (DEFAULT_TABLE table=test-table [id=901fa2a1bd2d46928f32659b99c2c92e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:25:23.962553 5408 tablet_service.cc:1467] Processing CreateTablet for tablet ffc93cd781b84dab89479648d921b944 (DEFAULT_TABLE table=test-table [id=901fa2a1bd2d46928f32659b99c2c92e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:25:23.962719 5277 tablet_service.cc:1467] Processing CreateTablet for tablet ffc93cd781b84dab89479648d921b944 (DEFAULT_TABLE table=test-table [id=901fa2a1bd2d46928f32659b99c2c92e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:25:23.963498 5146 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffc93cd781b84dab89479648d921b944. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:25:23.963539 5408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffc93cd781b84dab89479648d921b944. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:25:23.963539 5277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffc93cd781b84dab89479648d921b944. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:25:23.969514 5491 tablet_bootstrap.cc:492] T ffc93cd781b84dab89479648d921b944 P b311d398ae3242789dc406fcfa938cc6: Bootstrap starting.
I20260620 04:25:23.970899 5492 tablet_bootstrap.cc:492] T ffc93cd781b84dab89479648d921b944 P 738a393e040f4ce2bdd6c33fc5045a8a: Bootstrap starting.
I20260620 04:25:23.972083 5491 tablet_bootstrap.cc:654] T ffc93cd781b84dab89479648d921b944 P b311d398ae3242789dc406fcfa938cc6: Neither blocks nor log segments found. Creating new log.
I20260620 04:25:23.972249 5493 tablet_bootstrap.cc:492] T ffc93cd781b84dab89479648d921b944 P dfe9bbabeffd4361af69b770a5b530ac: Bootstrap starting.
I20260620 04:25:23.972951 5492 tablet_bootstrap.cc:654] T ffc93cd781b84dab89479648d921b944 P 738a393e040f4ce2bdd6c33fc5045a8a: Neither blocks nor log segments found. Creating new log.
I20260620 04:25:23.972958 5491 log.cc:824] T ffc93cd781b84dab89479648d921b944 P b311d398ae3242789dc406fcfa938cc6: Log is configured to *not* fsync() on all Append() calls
I20260620 04:25:23.973659 5492 log.cc:824] T ffc93cd781b84dab89479648d921b944 P 738a393e040f4ce2bdd6c33fc5045a8a: Log is configured to *not* fsync() on all Append() calls
I20260620 04:25:23.974211 5493 tablet_bootstrap.cc:654] T ffc93cd781b84dab89479648d921b944 P dfe9bbabeffd4361af69b770a5b530ac: Neither blocks nor log segments found. Creating new log.
I20260620 04:25:23.974817 5493 log.cc:824] T ffc93cd781b84dab89479648d921b944 P dfe9bbabeffd4361af69b770a5b530ac: Log is configured to *not* fsync() on all Append() calls
I20260620 04:25:23.974938 5491 tablet_bootstrap.cc:492] T ffc93cd781b84dab89479648d921b944 P b311d398ae3242789dc406fcfa938cc6: No bootstrap required, opened a new log
I20260620 04:25:23.975124 5491 ts_tablet_manager.cc:1397] T ffc93cd781b84dab89479648d921b944 P b311d398ae3242789dc406fcfa938cc6: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20260620 04:25:23.975564 5492 tablet_bootstrap.cc:492] T ffc93cd781b84dab89479648d921b944 P 738a393e040f4ce2bdd6c33fc5045a8a: No bootstrap required, opened a new log
I20260620 04:25:23.975744 5492 ts_tablet_manager.cc:1397] T ffc93cd781b84dab89479648d921b944 P 738a393e040f4ce2bdd6c33fc5045a8a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 04:25:23.976387 5493 tablet_bootstrap.cc:492] T ffc93cd781b84dab89479648d921b944 P dfe9bbabeffd4361af69b770a5b530ac: No bootstrap required, opened a new log
I20260620 04:25:23.976547 5493 ts_tablet_manager.cc:1397] T ffc93cd781b84dab89479648d921b944 P dfe9bbabeffd4361af69b770a5b530ac: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260620 04:25:23.979506 5491 raft_consensus.cc:348] T ffc93cd781b84dab89479648d921b944 P b311d398ae3242789dc406fcfa938cc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "738a393e040f4ce2bdd6c33fc5045a8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38441 } } peers { permanent_uuid: "b311d398ae3242789dc406fcfa938cc6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39687 } } peers { permanent_uuid: "dfe9bbabeffd4361af69b770a5b530ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34415 } }
I20260620 04:25:23.979768 5491 raft_consensus.cc:374] T ffc93cd781b84dab89479648d921b944 P b311d398ae3242789dc406fcfa938cc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:25:23.979825 5491 raft_consensus.cc:729] T ffc93cd781b84dab89479648d921b944 P b311d398ae3242789dc406fcfa938cc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b311d398ae3242789dc406fcfa938cc6, State: Initialized, Role: FOLLOWER
I20260620 04:25:23.979805 5492 raft_consensus.cc:348] T ffc93cd781b84dab89479648d921b944 P 738a393e040f4ce2bdd6c33fc5045a8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "738a393e040f4ce2bdd6c33fc5045a8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38441 } } peers { permanent_uuid: "b311d398ae3242789dc406fcfa938cc6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39687 } } peers { permanent_uuid: "dfe9bbabeffd4361af69b770a5b530ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34415 } }
I20260620 04:25:23.979797 5493 raft_consensus.cc:348] T ffc93cd781b84dab89479648d921b944 P dfe9bbabeffd4361af69b770a5b530ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "738a393e040f4ce2bdd6c33fc5045a8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38441 } } peers { permanent_uuid: "b311d398ae3242789dc406fcfa938cc6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39687 } } peers { permanent_uuid: "dfe9bbabeffd4361af69b770a5b530ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34415 } }
I20260620 04:25:23.980057 5492 raft_consensus.cc:374] T ffc93cd781b84dab89479648d921b944 P 738a393e040f4ce2bdd6c33fc5045a8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:25:23.980083 5493 raft_consensus.cc:374] T ffc93cd781b84dab89479648d921b944 P dfe9bbabeffd4361af69b770a5b530ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:25:23.980103 5492 raft_consensus.cc:729] T ffc93cd781b84dab89479648d921b944 P 738a393e040f4ce2bdd6c33fc5045a8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 738a393e040f4ce2bdd6c33fc5045a8a, State: Initialized, Role: FOLLOWER
I20260620 04:25:23.980140 5493 raft_consensus.cc:729] T ffc93cd781b84dab89479648d921b944 P dfe9bbabeffd4361af69b770a5b530ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfe9bbabeffd4361af69b770a5b530ac, State: Initialized, Role: FOLLOWER
I20260620 04:25:23.980301 5491 consensus_queue.cc:260] T ffc93cd781b84dab89479648d921b944 P b311d398ae3242789dc406fcfa938cc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "738a393e040f4ce2bdd6c33fc5045a8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38441 } } peers { permanent_uuid: "b311d398ae3242789dc406fcfa938cc6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39687 } } peers { permanent_uuid: "dfe9bbabeffd4361af69b770a5b530ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34415 } }
I20260620 04:25:23.980463 5492 consensus_queue.cc:260] T ffc93cd781b84dab89479648d921b944 P 738a393e040f4ce2bdd6c33fc5045a8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "738a393e040f4ce2bdd6c33fc5045a8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38441 } } peers { permanent_uuid: "b311d398ae3242789dc406fcfa938cc6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39687 } } peers { permanent_uuid: "dfe9bbabeffd4361af69b770a5b530ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34415 } }
I20260620 04:25:23.980551 5493 consensus_queue.cc:260] T ffc93cd781b84dab89479648d921b944 P dfe9bbabeffd4361af69b770a5b530ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "738a393e040f4ce2bdd6c33fc5045a8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38441 } } peers { permanent_uuid: "b311d398ae3242789dc406fcfa938cc6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39687 } } peers { permanent_uuid: "dfe9bbabeffd4361af69b770a5b530ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34415 } }
I20260620 04:25:23.981009 5212 heartbeater.cc:503] Master 127.0.98.62:44867 was elected leader, sending a full tablet report...
I20260620 04:25:23.981254 5491 ts_tablet_manager.cc:1428] T ffc93cd781b84dab89479648d921b944 P b311d398ae3242789dc406fcfa938cc6: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260620 04:25:23.981393 5493 ts_tablet_manager.cc:1428] T ffc93cd781b84dab89479648d921b944 P dfe9bbabeffd4361af69b770a5b530ac: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 04:25:23.981720 5343 heartbeater.cc:503] Master 127.0.98.62:44867 was elected leader, sending a full tablet report...
I20260620 04:25:23.981866 5474 heartbeater.cc:503] Master 127.0.98.62:44867 was elected leader, sending a full tablet report...
I20260620 04:25:23.982331 5492 ts_tablet_manager.cc:1428] T ffc93cd781b84dab89479648d921b944 P 738a393e040f4ce2bdd6c33fc5045a8a: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
W20260620 04:25:24.008392 5344 tablet.cc:2369] T ffc93cd781b84dab89479648d921b944 P dfe9bbabeffd4361af69b770a5b530ac: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 04:25:24.096292 5213 tablet.cc:2369] T ffc93cd781b84dab89479648d921b944 P b311d398ae3242789dc406fcfa938cc6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 04:25:24.169817 5475 tablet.cc:2369] T ffc93cd781b84dab89479648d921b944 P 738a393e040f4ce2bdd6c33fc5045a8a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 04:25:24.179291 5497 raft_consensus.cc:482] T ffc93cd781b84dab89479648d921b944 P b311d398ae3242789dc406fcfa938cc6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:25:24.179528 5497 raft_consensus.cc:504] T ffc93cd781b84dab89479648d921b944 P b311d398ae3242789dc406fcfa938cc6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "738a393e040f4ce2bdd6c33fc5045a8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38441 } } peers { permanent_uuid: "b311d398ae3242789dc406fcfa938cc6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39687 } } peers { permanent_uuid: "dfe9bbabeffd4361af69b770a5b530ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34415 } }
I20260620 04:25:24.180845 5497 leader_election.cc:288] T ffc93cd781b84dab89479648d921b944 P b311d398ae3242789dc406fcfa938cc6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 738a393e040f4ce2bdd6c33fc5045a8a (127.0.98.3:38441), dfe9bbabeffd4361af69b770a5b530ac (127.0.98.2:34415)
I20260620 04:25:24.183014 5486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:24.181248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37517 (local address 127.0.98.3:38441)
0620 04:25:24.181574 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:24.181579 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:24.181593 (+ 14us) server_negotiation.cc:407] Connection header received
0620 04:25:24.181632 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:24.181635 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:24.181678 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:24.181749 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:24.182531 (+ 782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:24.182535 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:24.182538 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:25:24.182554 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:24.182577 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:25:24.182587 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:24.182654 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:24.182836 (+ 182us) server_negotiation.cc:299] Negotiation successful
0620 04:25:24.182881 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231}
I20260620 04:25:24.183147 5488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:24.181564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59107 (local address 127.0.98.2:34415)
0620 04:25:24.181705 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:24.181708 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:24.181720 (+ 12us) server_negotiation.cc:407] Connection header received
0620 04:25:24.181768 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:24.181771 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:24.181807 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:24.181893 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:24.182691 (+ 798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:24.182695 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:24.182698 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:25:24.182712 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:24.182733 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:25:24.182744 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:24.182827 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:24.182967 (+ 140us) server_negotiation.cc:299] Negotiation successful
0620 04:25:24.183015 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 04:25:24.183029 5500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:24.181080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38441 (local address 127.0.98.1:37517)
0620 04:25:24.181313 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:24.181328 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:24.181422 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:24.181788 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:24.181790 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:24.181806 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:24.182328 (+ 522us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 04:25:24.182331 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:24.182345 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 04:25:24.182377 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:25:24.182726 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:25:24.182729 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:25:24.182732 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 04:25:24.182816 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 04:25:24.182855 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":83,"threads_started":1}
I20260620 04:25:24.183480 5501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:24.181119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34415 (local address 127.0.98.1:59107)
0620 04:25:24.181423 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:24.181435 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:24.181536 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:24.181922 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:24.181925 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:24.181939 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:24.182524 (+ 585us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 04:25:24.182526 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:24.182537 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 04:25:24.182563 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:25:24.182864 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:25:24.182867 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:25:24.182870 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 04:25:24.183286 (+ 416us) client_negotiation.cc:240] Negotiation successful
0620 04:25:24.183322 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":97,"threads_started":1}
I20260620 04:25:24.184168 5297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffc93cd781b84dab89479648d921b944" candidate_uuid: "b311d398ae3242789dc406fcfa938cc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfe9bbabeffd4361af69b770a5b530ac" is_pre_election: true
I20260620 04:25:24.184168 5428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffc93cd781b84dab89479648d921b944" candidate_uuid: "b311d398ae3242789dc406fcfa938cc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "738a393e040f4ce2bdd6c33fc5045a8a" is_pre_election: true
I20260620 04:25:24.184489 5428 raft_consensus.cc:2454] T ffc93cd781b84dab89479648d921b944 P 738a393e040f4ce2bdd6c33fc5045a8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b311d398ae3242789dc406fcfa938cc6 in term 0.
I20260620 04:25:24.184485 5297 raft_consensus.cc:2454] T ffc93cd781b84dab89479648d921b944 P dfe9bbabeffd4361af69b770a5b530ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b311d398ae3242789dc406fcfa938cc6 in term 0.
I20260620 04:25:24.184974 5100 leader_election.cc:302] T ffc93cd781b84dab89479648d921b944 P b311d398ae3242789dc406fcfa938cc6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b311d398ae3242789dc406fcfa938cc6, dfe9bbabeffd4361af69b770a5b530ac; no voters:
I20260620 04:25:24.185246 5497 raft_consensus.cc:2781] T ffc93cd781b84dab89479648d921b944 P b311d398ae3242789dc406fcfa938cc6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 04:25:24.185334 5497 raft_consensus.cc:482] T ffc93cd781b84dab89479648d921b944 P b311d398ae3242789dc406fcfa938cc6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 04:25:24.185386 5497 raft_consensus.cc:3037] T ffc93cd781b84dab89479648d921b944 P b311d398ae3242789dc406fcfa938cc6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:25:24.186697 5497 raft_consensus.cc:504] T ffc93cd781b84dab89479648d921b944 P b311d398ae3242789dc406fcfa938cc6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "738a393e040f4ce2bdd6c33fc5045a8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38441 } } peers { permanent_uuid: "b311d398ae3242789dc406fcfa938cc6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39687 } } peers { permanent_uuid: "dfe9bbabeffd4361af69b770a5b530ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34415 } }
I20260620 04:25:24.187085 5497 leader_election.cc:288] T ffc93cd781b84dab89479648d921b944 P b311d398ae3242789dc406fcfa938cc6 [CANDIDATE]: Term 1 election: Requested vote from peers 738a393e040f4ce2bdd6c33fc5045a8a (127.0.98.3:38441), dfe9bbabeffd4361af69b770a5b530ac (127.0.98.2:34415)
I20260620 04:25:24.192054 5297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffc93cd781b84dab89479648d921b944" candidate_uuid: "b311d398ae3242789dc406fcfa938cc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfe9bbabeffd4361af69b770a5b530ac"
I20260620 04:25:24.192194 5297 raft_consensus.cc:3037] T ffc93cd781b84dab89479648d921b944 P dfe9bbabeffd4361af69b770a5b530ac [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:25:24.192476 5428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffc93cd781b84dab89479648d921b944" candidate_uuid: "b311d398ae3242789dc406fcfa938cc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "738a393e040f4ce2bdd6c33fc5045a8a"
I20260620 04:25:24.192606 5428 raft_consensus.cc:3037] T ffc93cd781b84dab89479648d921b944 P 738a393e040f4ce2bdd6c33fc5045a8a [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:25:24.193586 5297 raft_consensus.cc:2454] T ffc93cd781b84dab89479648d921b944 P dfe9bbabeffd4361af69b770a5b530ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b311d398ae3242789dc406fcfa938cc6 in term 1.
I20260620 04:25:24.193758 5428 raft_consensus.cc:2454] T ffc93cd781b84dab89479648d921b944 P 738a393e040f4ce2bdd6c33fc5045a8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b311d398ae3242789dc406fcfa938cc6 in term 1.
I20260620 04:25:24.193939 5100 leader_election.cc:302] T ffc93cd781b84dab89479648d921b944 P b311d398ae3242789dc406fcfa938cc6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b311d398ae3242789dc406fcfa938cc6, dfe9bbabeffd4361af69b770a5b530ac; no voters:
I20260620 04:25:24.194166 5497 raft_consensus.cc:2781] T ffc93cd781b84dab89479648d921b944 P b311d398ae3242789dc406fcfa938cc6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:25:24.194420 5497 raft_consensus.cc:686] T ffc93cd781b84dab89479648d921b944 P b311d398ae3242789dc406fcfa938cc6 [term 1 LEADER]: Becoming Leader. State: Replica: b311d398ae3242789dc406fcfa938cc6, State: Running, Role: LEADER
I20260620 04:25:24.194785 5497 consensus_queue.cc:237] T ffc93cd781b84dab89479648d921b944 P b311d398ae3242789dc406fcfa938cc6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "738a393e040f4ce2bdd6c33fc5045a8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38441 } } peers { permanent_uuid: "b311d398ae3242789dc406fcfa938cc6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39687 } } peers { permanent_uuid: "dfe9bbabeffd4361af69b770a5b530ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34415 } }
I20260620 04:25:24.198155 5022 catalog_manager.cc:5495] T ffc93cd781b84dab89479648d921b944 P b311d398ae3242789dc406fcfa938cc6 reported cstate change: term changed from 0 to 1, leader changed from <none> to b311d398ae3242789dc406fcfa938cc6 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "b311d398ae3242789dc406fcfa938cc6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "738a393e040f4ce2bdd6c33fc5045a8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38441 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b311d398ae3242789dc406fcfa938cc6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dfe9bbabeffd4361af69b770a5b530ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34415 } health_report { overall_health: UNKNOWN } } }
I20260620 04:25:24.255522 5490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:24.254258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45356 (local address 127.0.98.1:39687)
0620 04:25:24.254372 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:24.254375 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:24.254388 (+ 13us) server_negotiation.cc:407] Connection header received
0620 04:25:24.254463 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:24.254466 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:24.254529 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:24.254599 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:24.255210 (+ 611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:24.255213 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:24.255216 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:25:24.255227 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:24.255248 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:25:24.255256 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:24.255335 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:24.255381 (+ 46us) server_negotiation.cc:299] Negotiation successful
0620 04:25:24.255414 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 04:25:24.261606 5428 raft_consensus.cc:1261] T ffc93cd781b84dab89479648d921b944 P 738a393e040f4ce2bdd6c33fc5045a8a [term 1 FOLLOWER]: Refusing update from remote peer b311d398ae3242789dc406fcfa938cc6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:25:24.261610 5297 raft_consensus.cc:1261] T ffc93cd781b84dab89479648d921b944 P dfe9bbabeffd4361af69b770a5b530ac [term 1 FOLLOWER]: Refusing update from remote peer b311d398ae3242789dc406fcfa938cc6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:25:24.262250 5502 consensus_queue.cc:1035] T ffc93cd781b84dab89479648d921b944 P b311d398ae3242789dc406fcfa938cc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "738a393e040f4ce2bdd6c33fc5045a8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:25:24.262421 5497 consensus_queue.cc:1035] T ffc93cd781b84dab89479648d921b944 P b311d398ae3242789dc406fcfa938cc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfe9bbabeffd4361af69b770a5b530ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:25:24.270382 5506 mvcc.cc:205] Tried to move back new op lower bound from 7298783331368001536 to 7298783331104858112. Current Snapshot: MvccSnapshot[applied={T|T < 7298783331368001536}]
I20260620 04:25:24.270622 5507 mvcc.cc:205] Tried to move back new op lower bound from 7298783331368001536 to 7298783331104858112. Current Snapshot: MvccSnapshot[applied={T|T < 7298783331368001536}]
I20260620 04:25:24.278466 5022 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47038:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:25:24.278620 5022 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47038:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:25:24.280795 5022 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 59aaa1415d1a45d58583d019b6a650a1: Sending DeleteTablet for 3 replicas of tablet ffc93cd781b84dab89479648d921b944
I20260620 04:25:24.281620 5146 tablet_service.cc:1514] Processing DeleteTablet for tablet ffc93cd781b84dab89479648d921b944 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:25:24 UTC) from {username='slave'} at 127.0.0.1:45348
I20260620 04:25:24.281661 5277 tablet_service.cc:1514] Processing DeleteTablet for tablet ffc93cd781b84dab89479648d921b944 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:25:24 UTC) from {username='slave'} at 127.0.0.1:51738
I20260620 04:25:24.281656 5408 tablet_service.cc:1514] Processing DeleteTablet for tablet ffc93cd781b84dab89479648d921b944 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:25:24 UTC) from {username='slave'} at 127.0.0.1:36840
I20260620 04:25:24.282085 5515 tablet_replica.cc:330] T ffc93cd781b84dab89479648d921b944 P b311d398ae3242789dc406fcfa938cc6: stopping tablet replica
I20260620 04:25:24.282393 5515 raft_consensus.cc:2229] T ffc93cd781b84dab89479648d921b944 P b311d398ae3242789dc406fcfa938cc6 [term 1 LEADER]: Raft consensus shutting down.
I20260620 04:25:24.282544 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 5084
I20260620 04:25:24.282761 5515 raft_consensus.cc:2258] T ffc93cd781b84dab89479648d921b944 P b311d398ae3242789dc406fcfa938cc6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:25:24.284657 5516 tablet_replica.cc:330] T ffc93cd781b84dab89479648d921b944 P 738a393e040f4ce2bdd6c33fc5045a8a: stopping tablet replica
I20260620 04:25:24.284931 5516 raft_consensus.cc:2229] T ffc93cd781b84dab89479648d921b944 P 738a393e040f4ce2bdd6c33fc5045a8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:25:24.285214 5516 raft_consensus.cc:2258] T ffc93cd781b84dab89479648d921b944 P 738a393e040f4ce2bdd6c33fc5045a8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:25:24.285521 5517 tablet_replica.cc:330] T ffc93cd781b84dab89479648d921b944 P dfe9bbabeffd4361af69b770a5b530ac: stopping tablet replica
I20260620 04:25:24.285789 5517 raft_consensus.cc:2229] T ffc93cd781b84dab89479648d921b944 P dfe9bbabeffd4361af69b770a5b530ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:25:24.286051 5517 raft_consensus.cc:2258] T ffc93cd781b84dab89479648d921b944 P dfe9bbabeffd4361af69b770a5b530ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:25:24.286711 5516 ts_tablet_manager.cc:1905] T ffc93cd781b84dab89479648d921b944 P 738a393e040f4ce2bdd6c33fc5045a8a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:25:24.287317 5517 ts_tablet_manager.cc:1905] T ffc93cd781b84dab89479648d921b944 P dfe9bbabeffd4361af69b770a5b530ac: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 04:25:24.288456 5008 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39687 (error 108)
I20260620 04:25:24.289278 5516 ts_tablet_manager.cc:1918] T ffc93cd781b84dab89479648d921b944 P 738a393e040f4ce2bdd6c33fc5045a8a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:25:24.289350 5487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:24.289088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39687 (local address 127.0.0.1:45368)
0620 04:25:24.289207 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:24.289271 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260620 04:25:24.289367 5516 log.cc:1196] T ffc93cd781b84dab89479648d921b944 P 738a393e040f4ce2bdd6c33fc5045a8a: Deleting WAL directory at /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929496330398-392-0/minicluster-data/ts-2/wal/wals/ffc93cd781b84dab89479648d921b944
I20260620 04:25:24.289467 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 5215
I20260620 04:25:24.289644 5516 ts_tablet_manager.cc:1939] T ffc93cd781b84dab89479648d921b944 P 738a393e040f4ce2bdd6c33fc5045a8a: Deleting consensus metadata
W20260620 04:25:24.292557 5008 catalog_manager.cc:4584] TS b311d398ae3242789dc406fcfa938cc6 (127.0.98.1:39687): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ffc93cd781b84dab89479648d921b944: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39687: connect: Connection refused (error 111)
I20260620 04:25:24.292769 5008 catalog_manager.cc:4841] TS 738a393e040f4ce2bdd6c33fc5045a8a (127.0.98.3:38441): tablet ffc93cd781b84dab89479648d921b944 (table test-table [id=901fa2a1bd2d46928f32659b99c2c92e]) successfully deleted
I20260620 04:25:24.294577 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 5346
I20260620 04:25:24.295848 5487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:24.294689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34415 (local address 127.0.0.1:51750)
0620 04:25:24.295720 (+ 1031us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:24.295772 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:34415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":952}
I20260620 04:25:24.299314 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 4993
2026-06-20T04:25:24Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1104 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260620 04:25:24.319161 392 mini_oidc.cc:152] Starting JWKS server
I20260620 04:25:24.320760 392 webserver.cc:460] Webserver started at https://127.0.0.1:38365/ using document root <none> and password file <none>
I20260620 04:25:24.320921 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 04:25:24.321174 392 webserver.cc:460] Webserver started at http://127.0.0.1:45349/ using document root <none> and password file <none>
2026-06-20T04:25:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:25:24Z Disabled control of system clock
I20260620 04:25:24.329439 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781929496330398-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781929496330398-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781929496330398-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41343
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:35151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41343
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38365/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781929496330398-392-0/testchainca.cert with env {}
W20260620 04:25:24.434794 5524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:25:24.435037 5524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:25:24.435074 5524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:25:24.439015 5524 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:25:24.439074 5524 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:25:24.439088 5524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:25:24.439103 5524 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:25:24.439116 5524 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 04:25:24.439131 5524 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 04:25:24.439142 5524 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38365/jwks.json
I20260620 04:25:24.444434 5524 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781929496330398-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781929496330398-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41343
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38365/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781929496330398-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5524
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781929496330398-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:25:24.445353 5524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:25:24.446286 5524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:25:24.453991 5529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:25:24.454059 5530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:25:24.454002 5532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:25:24.454389 5524 server_base.cc:1034] running on GCE node
I20260620 04:25:24.454905 5524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:25:24.455928 5524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:25:24.457100 5524 hybrid_clock.cc:648] HybridClock initialized: now 1781929524457075 us; error 36 us; skew 500 ppm
I20260620 04:25:24.458781 5524 webserver.cc:460] Webserver started at http://127.0.98.62:43363/ using document root <none> and password file <none>
I20260620 04:25:24.459301 5524 fs_manager.cc:359] Metadata directory not provided
I20260620 04:25:24.459362 5524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:25:24.459529 5524 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:25:24.461444 5524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781929496330398-392-0/minicluster-data/master-0/data/instance:
uuid: "763a5f74aee24453a6e822efe9ac1a73"
format_stamp: "Formatted at 2026-06-20 04:25:24 on dist-test-slave-nk6d"
server_key: "0a9a6c53b8957d8bf57ec29f87b41758"
server_key_iv: "4a1c5de4ffbfd7b2719c61dde4aed0fe"
server_key_version: "encryptionkey@0"
I20260620 04:25:24.461948 5524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781929496330398-392-0/minicluster-data/master-0/wal/instance:
uuid: "763a5f74aee24453a6e822efe9ac1a73"
format_stamp: "Formatted at 2026-06-20 04:25:24 on dist-test-slave-nk6d"
server_key: "0a9a6c53b8957d8bf57ec29f87b41758"
server_key_iv: "4a1c5de4ffbfd7b2719c61dde4aed0fe"
server_key_version: "encryptionkey@0"
I20260620 04:25:24.465313 5524 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 04:25:24.467551 5538 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:24.468755 5524 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:25:24.468860 5524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781929496330398-392-0/minicluster-data/master-0/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781929496330398-392-0/minicluster-data/master-0/wal
uuid: "763a5f74aee24453a6e822efe9ac1a73"
format_stamp: "Formatted at 2026-06-20 04:25:24 on dist-test-slave-nk6d"
server_key: "0a9a6c53b8957d8bf57ec29f87b41758"
server_key_iv: "4a1c5de4ffbfd7b2719c61dde4aed0fe"
server_key_version: "encryptionkey@0"
I20260620 04:25:24.468948 5524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781929496330398-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781929496330398-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781929496330398-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:25:24.485044 5524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:25:24.485705 5524 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:25:24.485874 5524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:25:24.492547 5524 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41343
I20260620 04:25:24.492590 5590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41343 every 8 connection(s)
I20260620 04:25:24.493474 5524 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:25:24.494652 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 5524
I20260620 04:25:24.494773 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781929496330398-392-0/minicluster-data/master-0/wal/instance
I20260620 04:25:24.495093 392 external_mini_cluster.cc:1428] Setting key 20b0467992bf57a1df54e8b5ad9e3d72
I20260620 04:25:24.497241 5591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:25:24.502683 5591 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 763a5f74aee24453a6e822efe9ac1a73: Bootstrap starting.
I20260620 04:25:24.503907 5594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:24.496506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33316 (local address 127.0.98.62:41343)
0620 04:25:24.497163 (+ 657us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:24.497178 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:24.497220 (+ 42us) server_negotiation.cc:407] Connection header received
0620 04:25:24.497897 (+ 677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:24.497916 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:24.498327 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:24.498640 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:24.499048 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:24.500055 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:24.500917 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:24.501393 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:24.501532 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:24.502313 (+ 781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:24.502318 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:24.502330 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:25:24.502693 (+ 363us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:24.502751 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:25:24.502776 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:24.502861 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:24.502966 (+ 105us) server_negotiation.cc:299] Negotiation successful
0620 04:25:24.503226 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":139,"threads_started":1}
I20260620 04:25:24.504900 5591 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 763a5f74aee24453a6e822efe9ac1a73: Neither blocks nor log segments found. Creating new log.
I20260620 04:25:24.505610 5591 log.cc:824] T 00000000000000000000000000000000 P 763a5f74aee24453a6e822efe9ac1a73: Log is configured to *not* fsync() on all Append() calls
I20260620 04:25:24.506553 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 04:25:24.507602 5591 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 763a5f74aee24453a6e822efe9ac1a73: No bootstrap required, opened a new log
I20260620 04:25:24.511026 5591 raft_consensus.cc:348] T 00000000000000000000000000000000 P 763a5f74aee24453a6e822efe9ac1a73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "763a5f74aee24453a6e822efe9ac1a73" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41343 } }
I20260620 04:25:24.511274 5591 raft_consensus.cc:374] T 00000000000000000000000000000000 P 763a5f74aee24453a6e822efe9ac1a73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:25:24.511308 5591 raft_consensus.cc:729] T 00000000000000000000000000000000 P 763a5f74aee24453a6e822efe9ac1a73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 763a5f74aee24453a6e822efe9ac1a73, State: Initialized, Role: FOLLOWER
I20260620 04:25:24.511732 5591 consensus_queue.cc:260] T 00000000000000000000000000000000 P 763a5f74aee24453a6e822efe9ac1a73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "763a5f74aee24453a6e822efe9ac1a73" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41343 } }
I20260620 04:25:24.511850 5591 raft_consensus.cc:388] T 00000000000000000000000000000000 P 763a5f74aee24453a6e822efe9ac1a73 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:25:24.511909 5591 raft_consensus.cc:482] T 00000000000000000000000000000000 P 763a5f74aee24453a6e822efe9ac1a73 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:25:24.512015 5591 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 763a5f74aee24453a6e822efe9ac1a73 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:25:24.512952 5591 raft_consensus.cc:504] T 00000000000000000000000000000000 P 763a5f74aee24453a6e822efe9ac1a73 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "763a5f74aee24453a6e822efe9ac1a73" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41343 } }
I20260620 04:25:24.513304 5591 leader_election.cc:302] T 00000000000000000000000000000000 P 763a5f74aee24453a6e822efe9ac1a73 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 763a5f74aee24453a6e822efe9ac1a73; no voters:
I20260620 04:25:24.513655 5591 leader_election.cc:288] T 00000000000000000000000000000000 P 763a5f74aee24453a6e822efe9ac1a73 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:25:24.513757 5596 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 763a5f74aee24453a6e822efe9ac1a73 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:25:24.513973 5596 raft_consensus.cc:686] T 00000000000000000000000000000000 P 763a5f74aee24453a6e822efe9ac1a73 [term 1 LEADER]: Becoming Leader. State: Replica: 763a5f74aee24453a6e822efe9ac1a73, State: Running, Role: LEADER
I20260620 04:25:24.514328 5596 consensus_queue.cc:237] T 00000000000000000000000000000000 P 763a5f74aee24453a6e822efe9ac1a73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "763a5f74aee24453a6e822efe9ac1a73" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41343 } }
I20260620 04:25:24.514873 5591 sys_catalog.cc:564] T 00000000000000000000000000000000 P 763a5f74aee24453a6e822efe9ac1a73 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:25:24.515874 5598 sys_catalog.cc:455] T 00000000000000000000000000000000 P 763a5f74aee24453a6e822efe9ac1a73 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 763a5f74aee24453a6e822efe9ac1a73. Latest consensus state: current_term: 1 leader_uuid: "763a5f74aee24453a6e822efe9ac1a73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "763a5f74aee24453a6e822efe9ac1a73" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41343 } } }
I20260620 04:25:24.516028 5598 sys_catalog.cc:458] T 00000000000000000000000000000000 P 763a5f74aee24453a6e822efe9ac1a73 [sys.catalog]: This master's current role is: LEADER
I20260620 04:25:24.516220 5597 sys_catalog.cc:455] T 00000000000000000000000000000000 P 763a5f74aee24453a6e822efe9ac1a73 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "763a5f74aee24453a6e822efe9ac1a73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "763a5f74aee24453a6e822efe9ac1a73" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41343 } } }
I20260620 04:25:24.516374 5597 sys_catalog.cc:458] T 00000000000000000000000000000000 P 763a5f74aee24453a6e822efe9ac1a73 [sys.catalog]: This master's current role is: LEADER
I20260620 04:25:24.516713 5605 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:25:24.519469 5605 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:25:24.524930 5605 catalog_manager.cc:1347] Generated new cluster ID: 547ceab47efc4c6985da1cbab6a25e41
I20260620 04:25:24.525002 5605 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:25:24.534611 5605 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:25:24.535822 5605 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:25:24.547796 5605 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 763a5f74aee24453a6e822efe9ac1a73: Generated new TSK 0
I20260620 04:25:24.548642 5605 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 04:25:24.582973 5621 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260620 04:25:24.583094 5621 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 04:25:24.583585 5594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:24.566801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33320 (local address 127.0.98.62:41343)
0620 04:25:24.566926 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:24.566931 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:24.567214 (+ 283us) server_negotiation.cc:407] Connection header received
0620 04:25:24.567358 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:24.567363 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:24.567414 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:24.567506 (+ 92us) server_negotiation.cc:226] Negotiated authn=JWT
0620 04:25:24.567757 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:24.568345 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:24.569571 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:24.569814 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:24.569887 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:24.570215 (+ 328us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 04:25:24.583245 (+ 13030us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38365/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 04:25:24.583476 (+ 231us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33320: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38365/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":46}
W20260620 04:25:24.583647 5594 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33320: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38365/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260620 04:25:24.585083 5620 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:25:24.566705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41343 (local address 127.0.0.1:33320)
0620 04:25:24.567056 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:24.567071 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:24.567195 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:24.567536 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:24.567539 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:24.567547 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0620 04:25:24.567653 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:24.567659 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:24.568445 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:24.568448 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:24.569456 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:24.569465 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:24.569892 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:24.569910 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:24.569995 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:24.570053 (+ 58us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 04:25:24.583485 (+ 13432us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38365/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 04:25:24.583748 (+ 263us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:41343: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38365/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":96,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:41343: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38365/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260620 04:25:24.585865 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 5524
2026-06-20T04:25:24Z chronyd exiting
I20260620 04:25:24.622200 392 test_util.cc:175] -----------------------------------------------
I20260620 04:25:24.622287 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781929496330398-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (304 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260620 04:25:24.623255 392 mini_oidc.cc:152] Starting JWKS server
I20260620 04:25:24.624120 392 webserver.cc:460] Webserver started at https://127.0.0.1:38883/ using document root <none> and password file <none>
I20260620 04:25:24.624208 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 04:25:24.624444 392 webserver.cc:460] Webserver started at http://127.0.0.1:41733/ using document root <none> and password file <none>
2026-06-20T04:25:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:25:24Z Disabled control of system clock
I20260620 04:25:24.632375 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781929496330398-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781929496330398-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781929496330398-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41499
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:43463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41499
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38883/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781929496330398-392-0/testchainca.cert with env {}
W20260620 04:25:24.737074 5627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:25:24.737339 5627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:25:24.737377 5627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:25:24.741272 5627 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:25:24.741321 5627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:25:24.741335 5627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:25:24.741349 5627 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:25:24.741362 5627 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 04:25:24.741376 5627 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 04:25:24.741389 5627 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38883/jwks.json
I20260620 04:25:24.746589 5627 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781929496330398-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781929496330398-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41499
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41499
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38883/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781929496330398-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5627
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781929496330398-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:25:24.747550 5627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:25:24.748505 5627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:25:24.756428 5632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:25:24.756481 5635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:25:24.756497 5627 server_base.cc:1034] running on GCE node
W20260620 04:25:24.756731 5633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:25:24.757261 5627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:25:24.758335 5627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:25:24.759506 5627 hybrid_clock.cc:648] HybridClock initialized: now 1781929524759488 us; error 29 us; skew 500 ppm
I20260620 04:25:24.761361 5627 webserver.cc:460] Webserver started at http://127.0.98.62:41279/ using document root <none> and password file <none>
I20260620 04:25:24.761911 5627 fs_manager.cc:359] Metadata directory not provided
I20260620 04:25:24.761977 5627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:25:24.762167 5627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:25:24.764276 5627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781929496330398-392-0/minicluster-data/master-0/data/instance:
uuid: "e260cc2882b14b34a76fe9dcc49035a6"
format_stamp: "Formatted at 2026-06-20 04:25:24 on dist-test-slave-nk6d"
server_key: "16e51977dd1b27c7d814000ddf3f3dbd"
server_key_iv: "404e720e32614953949c25ff1915cd86"
server_key_version: "encryptionkey@0"
I20260620 04:25:24.764804 5627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781929496330398-392-0/minicluster-data/master-0/wal/instance:
uuid: "e260cc2882b14b34a76fe9dcc49035a6"
format_stamp: "Formatted at 2026-06-20 04:25:24 on dist-test-slave-nk6d"
server_key: "16e51977dd1b27c7d814000ddf3f3dbd"
server_key_iv: "404e720e32614953949c25ff1915cd86"
server_key_version: "encryptionkey@0"
I20260620 04:25:24.768299 5627 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 04:25:24.770574 5641 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:24.771785 5627 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:25:24.771912 5627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781929496330398-392-0/minicluster-data/master-0/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781929496330398-392-0/minicluster-data/master-0/wal
uuid: "e260cc2882b14b34a76fe9dcc49035a6"
format_stamp: "Formatted at 2026-06-20 04:25:24 on dist-test-slave-nk6d"
server_key: "16e51977dd1b27c7d814000ddf3f3dbd"
server_key_iv: "404e720e32614953949c25ff1915cd86"
server_key_version: "encryptionkey@0"
I20260620 04:25:24.772033 5627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781929496330398-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781929496330398-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781929496330398-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:25:24.785876 5627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:25:24.786552 5627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:25:24.786727 5627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:25:24.793087 5627 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41499
I20260620 04:25:24.793125 5693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41499 every 8 connection(s)
I20260620 04:25:24.793983 5627 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:25:24.796952 5694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:25:24.797389 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 5627
I20260620 04:25:24.797511 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781929496330398-392-0/minicluster-data/master-0/wal/instance
I20260620 04:25:24.797813 392 external_mini_cluster.cc:1428] Setting key 3ccf335df7310dedf23e2a27f5151797
I20260620 04:25:24.802783 5694 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e260cc2882b14b34a76fe9dcc49035a6: Bootstrap starting.
I20260620 04:25:24.805274 5694 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e260cc2882b14b34a76fe9dcc49035a6: Neither blocks nor log segments found. Creating new log.
I20260620 04:25:24.806034 5694 log.cc:824] T 00000000000000000000000000000000 P e260cc2882b14b34a76fe9dcc49035a6: Log is configured to *not* fsync() on all Append() calls
I20260620 04:25:24.806797 5697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:24.799155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48854 (local address 127.0.98.62:41499)
0620 04:25:24.799753 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:24.799766 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:24.799801 (+ 35us) server_negotiation.cc:407] Connection header received
0620 04:25:24.800487 (+ 686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:24.800504 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:24.800896 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:24.801183 (+ 287us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:24.801502 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:24.802571 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:24.803728 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:24.804242 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:24.804365 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:24.805120 (+ 755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:24.805124 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:24.805137 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:25:24.805543 (+ 406us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:24.805597 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:25:24.805616 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:24.805689 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:24.805832 (+ 143us) server_negotiation.cc:299] Negotiation successful
0620 04:25:24.806082 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":98,"threads_started":1}
I20260620 04:25:24.807786 5694 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e260cc2882b14b34a76fe9dcc49035a6: No bootstrap required, opened a new log
I20260620 04:25:24.809376 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 04:25:24.811396 5694 raft_consensus.cc:348] T 00000000000000000000000000000000 P e260cc2882b14b34a76fe9dcc49035a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e260cc2882b14b34a76fe9dcc49035a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41499 } }
I20260620 04:25:24.811607 5694 raft_consensus.cc:374] T 00000000000000000000000000000000 P e260cc2882b14b34a76fe9dcc49035a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:25:24.811645 5694 raft_consensus.cc:729] T 00000000000000000000000000000000 P e260cc2882b14b34a76fe9dcc49035a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e260cc2882b14b34a76fe9dcc49035a6, State: Initialized, Role: FOLLOWER
I20260620 04:25:24.812021 5694 consensus_queue.cc:260] T 00000000000000000000000000000000 P e260cc2882b14b34a76fe9dcc49035a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e260cc2882b14b34a76fe9dcc49035a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41499 } }
I20260620 04:25:24.812139 5694 raft_consensus.cc:388] T 00000000000000000000000000000000 P e260cc2882b14b34a76fe9dcc49035a6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:25:24.812181 5694 raft_consensus.cc:482] T 00000000000000000000000000000000 P e260cc2882b14b34a76fe9dcc49035a6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:25:24.812254 5694 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e260cc2882b14b34a76fe9dcc49035a6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:25:24.813028 5694 raft_consensus.cc:504] T 00000000000000000000000000000000 P e260cc2882b14b34a76fe9dcc49035a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e260cc2882b14b34a76fe9dcc49035a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41499 } }
I20260620 04:25:24.813316 5694 leader_election.cc:302] T 00000000000000000000000000000000 P e260cc2882b14b34a76fe9dcc49035a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e260cc2882b14b34a76fe9dcc49035a6; no voters:
I20260620 04:25:24.813562 5694 leader_election.cc:288] T 00000000000000000000000000000000 P e260cc2882b14b34a76fe9dcc49035a6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:25:24.813711 5699 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e260cc2882b14b34a76fe9dcc49035a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:25:24.813905 5699 raft_consensus.cc:686] T 00000000000000000000000000000000 P e260cc2882b14b34a76fe9dcc49035a6 [term 1 LEADER]: Becoming Leader. State: Replica: e260cc2882b14b34a76fe9dcc49035a6, State: Running, Role: LEADER
I20260620 04:25:24.814170 5699 consensus_queue.cc:237] T 00000000000000000000000000000000 P e260cc2882b14b34a76fe9dcc49035a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e260cc2882b14b34a76fe9dcc49035a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41499 } }
I20260620 04:25:24.814626 5694 sys_catalog.cc:564] T 00000000000000000000000000000000 P e260cc2882b14b34a76fe9dcc49035a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:25:24.815663 5701 sys_catalog.cc:455] T 00000000000000000000000000000000 P e260cc2882b14b34a76fe9dcc49035a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e260cc2882b14b34a76fe9dcc49035a6. Latest consensus state: current_term: 1 leader_uuid: "e260cc2882b14b34a76fe9dcc49035a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e260cc2882b14b34a76fe9dcc49035a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41499 } } }
I20260620 04:25:24.815663 5700 sys_catalog.cc:455] T 00000000000000000000000000000000 P e260cc2882b14b34a76fe9dcc49035a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e260cc2882b14b34a76fe9dcc49035a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e260cc2882b14b34a76fe9dcc49035a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41499 } } }
I20260620 04:25:24.815804 5701 sys_catalog.cc:458] T 00000000000000000000000000000000 P e260cc2882b14b34a76fe9dcc49035a6 [sys.catalog]: This master's current role is: LEADER
I20260620 04:25:24.815809 5700 sys_catalog.cc:458] T 00000000000000000000000000000000 P e260cc2882b14b34a76fe9dcc49035a6 [sys.catalog]: This master's current role is: LEADER
I20260620 04:25:24.816311 5708 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:25:24.819240 5708 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:25:24.824613 5708 catalog_manager.cc:1347] Generated new cluster ID: 2f3fee1c2847453ab6cdc0b5650c8769
I20260620 04:25:24.824700 5708 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:25:24.834061 5708 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:25:24.835305 5708 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:25:24.851755 5708 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e260cc2882b14b34a76fe9dcc49035a6: Generated new TSK 0
I20260620 04:25:24.852479 5708 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 04:25:24.873013 5724 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260620 04:25:24.873108 5724 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 04:25:24.873749 5697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:24.858737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48866 (local address 127.0.98.62:41499)
0620 04:25:24.858843 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:24.858847 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:24.859005 (+ 158us) server_negotiation.cc:407] Connection header received
0620 04:25:24.859169 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:24.859171 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:24.859219 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:24.859309 (+ 90us) server_negotiation.cc:226] Negotiated authn=JWT
0620 04:25:24.859526 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:24.860113 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:24.860970 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:24.861197 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:24.861280 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:24.861562 (+ 282us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 04:25:24.873354 (+ 11792us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38883/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 04:25:24.873614 (+ 260us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48866: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38883/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":35}
W20260620 04:25:24.873814 5697 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48866: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38883/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260620 04:25:24.873838 5723 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:25:24.858658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41499 (local address 127.0.0.1:48866)
0620 04:25:24.858927 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:24.858939 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:24.859042 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:24.859315 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:24.859318 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:24.859325 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0620 04:25:24.859434 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:24.859440 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:24.860214 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:24.860218 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:24.860860 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:24.860868 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:24.861280 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:24.861299 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:24.861385 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:24.861408 (+ 23us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 04:25:24.873535 (+ 12127us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38883/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 04:25:24.873683 (+ 148us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:41499: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38883/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":106,"threads_started":1}
I20260620 04:25:24.874719 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 5627
2026-06-20T04:25:24Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (289 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260620 04:25:24.912925 392 mini_oidc.cc:152] Starting JWKS server
I20260620 04:25:24.913791 392 webserver.cc:460] Webserver started at https://127.0.0.1:45441/ using document root <none> and password file <none>
I20260620 04:25:24.913877 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 04:25:24.914232 392 webserver.cc:460] Webserver started at http://127.0.0.1:42967/ using document root <none> and password file <none>
2026-06-20T04:25:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:25:24Z Disabled control of system clock
I20260620 04:25:24.922407 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781929496330398-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781929496330398-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781929496330398-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38645
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:40463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38645
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45441/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 04:25:25.030618 5730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:25:25.030893 5730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:25:25.030932 5730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:25:25.034878 5730 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:25:25.034934 5730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:25:25.034950 5730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:25:25.034965 5730 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:25:25.034977 5730 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 04:25:25.034992 5730 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 04:25:25.035009 5730 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45441/jwks.json
I20260620 04:25:25.039731 5730 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781929496330398-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781929496330398-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38645
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38645
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45441/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5730
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781929496330398-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:25:25.040759 5730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:25:25.042059 5730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:25:25.050091 5736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:25:25.050097 5738 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:25:25.050187 5735 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:25:25.050158 5730 server_base.cc:1034] running on GCE node
I20260620 04:25:25.050844 5730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:25:25.051940 5730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:25:25.053138 5730 hybrid_clock.cc:648] HybridClock initialized: now 1781929525053124 us; error 46 us; skew 500 ppm
I20260620 04:25:25.054951 5730 webserver.cc:460] Webserver started at http://127.0.98.62:38207/ using document root <none> and password file <none>
I20260620 04:25:25.055464 5730 fs_manager.cc:359] Metadata directory not provided
I20260620 04:25:25.055521 5730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:25:25.055680 5730 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:25:25.057678 5730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781929496330398-392-0/minicluster-data/master-0/data/instance:
uuid: "b387fd1e12014e8d911a8d4c1cdb7ae6"
format_stamp: "Formatted at 2026-06-20 04:25:25 on dist-test-slave-nk6d"
server_key: "69af0b78df8eac48f66bd78b0093a26a"
server_key_iv: "cbec892dac8445f91c4d3ab278003c72"
server_key_version: "encryptionkey@0"
I20260620 04:25:25.058188 5730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781929496330398-392-0/minicluster-data/master-0/wal/instance:
uuid: "b387fd1e12014e8d911a8d4c1cdb7ae6"
format_stamp: "Formatted at 2026-06-20 04:25:25 on dist-test-slave-nk6d"
server_key: "69af0b78df8eac48f66bd78b0093a26a"
server_key_iv: "cbec892dac8445f91c4d3ab278003c72"
server_key_version: "encryptionkey@0"
I20260620 04:25:25.061668 5730 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 04:25:25.064036 5744 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:25.065228 5730 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:25:25.065332 5730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781929496330398-392-0/minicluster-data/master-0/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781929496330398-392-0/minicluster-data/master-0/wal
uuid: "b387fd1e12014e8d911a8d4c1cdb7ae6"
format_stamp: "Formatted at 2026-06-20 04:25:25 on dist-test-slave-nk6d"
server_key: "69af0b78df8eac48f66bd78b0093a26a"
server_key_iv: "cbec892dac8445f91c4d3ab278003c72"
server_key_version: "encryptionkey@0"
I20260620 04:25:25.065418 5730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781929496330398-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781929496330398-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781929496330398-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:25:25.081863 5730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:25:25.082496 5730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:25:25.082649 5730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:25:25.089219 5730 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38645
I20260620 04:25:25.089241 5796 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38645 every 8 connection(s)
I20260620 04:25:25.090116 5730 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:25:25.093160 5797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:25:25.097752 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 5730
I20260620 04:25:25.097853 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781929496330398-392-0/minicluster-data/master-0/wal/instance
I20260620 04:25:25.098088 392 external_mini_cluster.cc:1428] Setting key 43852152f5a48662dc41fda12ab98840
I20260620 04:25:25.098595 5797 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b387fd1e12014e8d911a8d4c1cdb7ae6: Bootstrap starting.
I20260620 04:25:25.101166 5797 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b387fd1e12014e8d911a8d4c1cdb7ae6: Neither blocks nor log segments found. Creating new log.
I20260620 04:25:25.102020 5797 log.cc:824] T 00000000000000000000000000000000 P b387fd1e12014e8d911a8d4c1cdb7ae6: Log is configured to *not* fsync() on all Append() calls
I20260620 04:25:25.104223 5797 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b387fd1e12014e8d911a8d4c1cdb7ae6: No bootstrap required, opened a new log
I20260620 04:25:25.108470 5797 raft_consensus.cc:348] T 00000000000000000000000000000000 P b387fd1e12014e8d911a8d4c1cdb7ae6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b387fd1e12014e8d911a8d4c1cdb7ae6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38645 } }
I20260620 04:25:25.108752 5797 raft_consensus.cc:374] T 00000000000000000000000000000000 P b387fd1e12014e8d911a8d4c1cdb7ae6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:25:25.108835 5797 raft_consensus.cc:729] T 00000000000000000000000000000000 P b387fd1e12014e8d911a8d4c1cdb7ae6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b387fd1e12014e8d911a8d4c1cdb7ae6, State: Initialized, Role: FOLLOWER
I20260620 04:25:25.108856 5800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:25.099452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33130 (local address 127.0.98.62:38645)
0620 04:25:25.100152 (+ 700us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:25.100179 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:25.100222 (+ 43us) server_negotiation.cc:407] Connection header received
0620 04:25:25.101018 (+ 796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:25.101042 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:25.101569 (+ 527us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:25.101943 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:25.102390 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:25.103793 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:25.104752 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:25.105432 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:25.105601 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:25.106560 (+ 959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:25.106567 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:25.106587 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:25:25.107118 (+ 531us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:25.107200 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:25:25.107231 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:25.107351 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:25.107539 (+ 188us) server_negotiation.cc:299] Negotiation successful
0620 04:25:25.107845 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":105,"threads_started":1}
I20260620 04:25:25.109285 5797 consensus_queue.cc:260] T 00000000000000000000000000000000 P b387fd1e12014e8d911a8d4c1cdb7ae6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b387fd1e12014e8d911a8d4c1cdb7ae6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38645 } }
I20260620 04:25:25.109419 5797 raft_consensus.cc:388] T 00000000000000000000000000000000 P b387fd1e12014e8d911a8d4c1cdb7ae6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:25:25.109483 5797 raft_consensus.cc:482] T 00000000000000000000000000000000 P b387fd1e12014e8d911a8d4c1cdb7ae6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:25:25.109577 5797 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b387fd1e12014e8d911a8d4c1cdb7ae6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:25:25.110495 5797 raft_consensus.cc:504] T 00000000000000000000000000000000 P b387fd1e12014e8d911a8d4c1cdb7ae6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b387fd1e12014e8d911a8d4c1cdb7ae6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38645 } }
I20260620 04:25:25.110859 5797 leader_election.cc:302] T 00000000000000000000000000000000 P b387fd1e12014e8d911a8d4c1cdb7ae6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b387fd1e12014e8d911a8d4c1cdb7ae6; no voters:
I20260620 04:25:25.111181 5797 leader_election.cc:288] T 00000000000000000000000000000000 P b387fd1e12014e8d911a8d4c1cdb7ae6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:25:25.111279 5802 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b387fd1e12014e8d911a8d4c1cdb7ae6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:25:25.111490 5802 raft_consensus.cc:686] T 00000000000000000000000000000000 P b387fd1e12014e8d911a8d4c1cdb7ae6 [term 1 LEADER]: Becoming Leader. State: Replica: b387fd1e12014e8d911a8d4c1cdb7ae6, State: Running, Role: LEADER
I20260620 04:25:25.111792 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 04:25:25.111810 5802 consensus_queue.cc:237] T 00000000000000000000000000000000 P b387fd1e12014e8d911a8d4c1cdb7ae6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b387fd1e12014e8d911a8d4c1cdb7ae6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38645 } }
I20260620 04:25:25.112519 5797 sys_catalog.cc:564] T 00000000000000000000000000000000 P b387fd1e12014e8d911a8d4c1cdb7ae6 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:25:25.113312 5804 sys_catalog.cc:455] T 00000000000000000000000000000000 P b387fd1e12014e8d911a8d4c1cdb7ae6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b387fd1e12014e8d911a8d4c1cdb7ae6. Latest consensus state: current_term: 1 leader_uuid: "b387fd1e12014e8d911a8d4c1cdb7ae6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b387fd1e12014e8d911a8d4c1cdb7ae6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38645 } } }
I20260620 04:25:25.113440 5804 sys_catalog.cc:458] T 00000000000000000000000000000000 P b387fd1e12014e8d911a8d4c1cdb7ae6 [sys.catalog]: This master's current role is: LEADER
I20260620 04:25:25.113708 5803 sys_catalog.cc:455] T 00000000000000000000000000000000 P b387fd1e12014e8d911a8d4c1cdb7ae6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b387fd1e12014e8d911a8d4c1cdb7ae6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b387fd1e12014e8d911a8d4c1cdb7ae6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38645 } } }
I20260620 04:25:25.113797 5803 sys_catalog.cc:458] T 00000000000000000000000000000000 P b387fd1e12014e8d911a8d4c1cdb7ae6 [sys.catalog]: This master's current role is: LEADER
W20260620 04:25:25.117141 5818 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b387fd1e12014e8d911a8d4c1cdb7ae6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 04:25:25.117208 5818 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 04:25:25.117314 5811 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:25:25.118178 5811 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:25:25.123278 5811 catalog_manager.cc:1347] Generated new cluster ID: 4c67eba4ae1642e187f388f11967fec4
I20260620 04:25:25.123342 5811 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:25:25.146835 5811 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:25:25.148070 5811 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:25:25.152644 5811 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b387fd1e12014e8d911a8d4c1cdb7ae6: Generated new TSK 0
I20260620 04:25:25.153302 5811 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 04:25:25.170859 5800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:25.165686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33132 (local address 127.0.98.62:38645)
0620 04:25:25.165805 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:25.165810 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:25.165912 (+ 102us) server_negotiation.cc:407] Connection header received
0620 04:25:25.166050 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:25.166054 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:25.166108 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:25.166219 (+ 111us) server_negotiation.cc:226] Negotiated authn=JWT
0620 04:25:25.166476 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:25.167301 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:25.168258 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:25.168609 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:25.168725 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:25.169028 (+ 303us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 04:25:25.170518 (+ 1490us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45441/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 04:25:25.170729 (+ 211us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33132: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45441/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":48}
W20260620 04:25:25.170926 5800 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33132: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45441/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260620 04:25:25.172611 5826 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:25:25.165576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38645 (local address 127.0.0.1:33132)
0620 04:25:25.165831 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:25.165843 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:25.165937 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:25.166246 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:25.166249 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:25.166256 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0620 04:25:25.166374 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:25.166380 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:25.167434 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:25.167437 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:25.168152 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:25.168163 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:25.168717 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:25.168743 (+ 26us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:25.168831 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:25.168857 (+ 26us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 04:25:25.170705 (+ 1848us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45441/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 04:25:25.172448 (+ 1743us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:38645: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45441/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":178,"mutex_wait_us":1512,"thread_start_us":90,"threads_started":1}
I20260620 04:25:25.173455 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 5730
E20260620 04:25:25.181377 5827 webserver.cc:533] Webserver: sslize failed:
E20260620 04:25:25.181442 5827 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-06-20T04:25:25Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (298 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-20T04:25:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:25:25Z Disabled control of system clock
I20260620 04:25:25.219071 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781929496330398-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781929496330398-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781929496330398-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34043
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:37061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34043
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781929496330398-392-0/testchainca.cert with env {}
W20260620 04:25:25.323213 5831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:25:25.323480 5831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:25:25.323534 5831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:25:25.327524 5831 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:25:25.327592 5831 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:25:25.327615 5831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:25:25.327634 5831 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:25:25.327654 5831 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 04:25:25.327679 5831 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 04:25:25.327705 5831 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260620 04:25:25.333037 5831 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781929496330398-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781929496330398-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34043
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34043
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781929496330398-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5831
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781929496330398-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:25:25.334069 5831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:25:25.335057 5831 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:25:25.343340 5837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:25:25.343502 5831 server_base.cc:1034] running on GCE node
W20260620 04:25:25.343322 5839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:25:25.343327 5836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:25:25.344250 5831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:25:25.345326 5831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:25:25.346482 5831 hybrid_clock.cc:648] HybridClock initialized: now 1781929525346465 us; error 30 us; skew 500 ppm
I20260620 04:25:25.352308 5831 webserver.cc:460] Webserver started at http://127.0.98.62:33661/ using document root <none> and password file <none>
I20260620 04:25:25.352886 5831 fs_manager.cc:359] Metadata directory not provided
I20260620 04:25:25.352952 5831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:25:25.353142 5831 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:25:25.355165 5831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781929496330398-392-0/minicluster-data/master-0/data/instance:
uuid: "9700a70f0a7f428fb066dfa050515af8"
format_stamp: "Formatted at 2026-06-20 04:25:25 on dist-test-slave-nk6d"
server_key: "ae208994a5c7b87bc202d9b86a1d922e"
server_key_iv: "53d23a9c935ed64ad253f59b9b7313d0"
server_key_version: "encryptionkey@0"
I20260620 04:25:25.355703 5831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781929496330398-392-0/minicluster-data/master-0/wal/instance:
uuid: "9700a70f0a7f428fb066dfa050515af8"
format_stamp: "Formatted at 2026-06-20 04:25:25 on dist-test-slave-nk6d"
server_key: "ae208994a5c7b87bc202d9b86a1d922e"
server_key_iv: "53d23a9c935ed64ad253f59b9b7313d0"
server_key_version: "encryptionkey@0"
I20260620 04:25:25.359334 5831 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 04:25:25.361690 5845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:25.362929 5831 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 04:25:25.363041 5831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781929496330398-392-0/minicluster-data/master-0/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781929496330398-392-0/minicluster-data/master-0/wal
uuid: "9700a70f0a7f428fb066dfa050515af8"
format_stamp: "Formatted at 2026-06-20 04:25:25 on dist-test-slave-nk6d"
server_key: "ae208994a5c7b87bc202d9b86a1d922e"
server_key_iv: "53d23a9c935ed64ad253f59b9b7313d0"
server_key_version: "encryptionkey@0"
I20260620 04:25:25.363140 5831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781929496330398-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781929496330398-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781929496330398-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:25:25.379277 5831 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:25:25.379987 5831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:25:25.380167 5831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:25:25.386641 5831 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34043
I20260620 04:25:25.386653 5897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34043 every 8 connection(s)
I20260620 04:25:25.387531 5831 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:25:25.390462 5898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:25:25.394238 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 5831
I20260620 04:25:25.394341 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781929496330398-392-0/minicluster-data/master-0/wal/instance
I20260620 04:25:25.394583 392 external_mini_cluster.cc:1428] Setting key 840aa3be8fed9251e828f3924037b804
I20260620 04:25:25.395884 5898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9700a70f0a7f428fb066dfa050515af8: Bootstrap starting.
I20260620 04:25:25.398422 5898 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9700a70f0a7f428fb066dfa050515af8: Neither blocks nor log segments found. Creating new log.
I20260620 04:25:25.399266 5898 log.cc:824] T 00000000000000000000000000000000 P 9700a70f0a7f428fb066dfa050515af8: Log is configured to *not* fsync() on all Append() calls
I20260620 04:25:25.401211 5898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9700a70f0a7f428fb066dfa050515af8: No bootstrap required, opened a new log
I20260620 04:25:25.404549 5901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:25.395911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60260 (local address 127.0.98.62:34043)
0620 04:25:25.396595 (+ 684us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:25.396613 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:25.396654 (+ 41us) server_negotiation.cc:407] Connection header received
0620 04:25:25.397444 (+ 790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:25.397466 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:25.397943 (+ 477us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:25.398286 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:25.398754 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:25.400031 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:25.400892 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:25.401566 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:25.401738 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:25.402440 (+ 702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:25.402446 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:25.402463 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:25:25.402957 (+ 494us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:25.403023 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:25:25.403047 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:25.403143 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:25.403318 (+ 175us) server_negotiation.cc:299] Negotiation successful
0620 04:25:25.403575 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":102,"threads_started":1}
I20260620 04:25:25.405428 5898 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9700a70f0a7f428fb066dfa050515af8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9700a70f0a7f428fb066dfa050515af8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34043 } }
I20260620 04:25:25.405682 5898 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9700a70f0a7f428fb066dfa050515af8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:25:25.405736 5898 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9700a70f0a7f428fb066dfa050515af8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9700a70f0a7f428fb066dfa050515af8, State: Initialized, Role: FOLLOWER
I20260620 04:25:25.406177 5898 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9700a70f0a7f428fb066dfa050515af8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9700a70f0a7f428fb066dfa050515af8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34043 } }
I20260620 04:25:25.406320 5898 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9700a70f0a7f428fb066dfa050515af8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:25:25.406399 5898 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9700a70f0a7f428fb066dfa050515af8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:25:25.406497 5898 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9700a70f0a7f428fb066dfa050515af8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:25:25.407475 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 04:25:25.407509 5898 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9700a70f0a7f428fb066dfa050515af8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9700a70f0a7f428fb066dfa050515af8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34043 } }
I20260620 04:25:25.407848 5898 leader_election.cc:302] T 00000000000000000000000000000000 P 9700a70f0a7f428fb066dfa050515af8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9700a70f0a7f428fb066dfa050515af8; no voters:
I20260620 04:25:25.408206 5898 leader_election.cc:288] T 00000000000000000000000000000000 P 9700a70f0a7f428fb066dfa050515af8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:25:25.408313 5903 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9700a70f0a7f428fb066dfa050515af8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:25:25.408537 5903 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9700a70f0a7f428fb066dfa050515af8 [term 1 LEADER]: Becoming Leader. State: Replica: 9700a70f0a7f428fb066dfa050515af8, State: Running, Role: LEADER
I20260620 04:25:25.408917 5903 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9700a70f0a7f428fb066dfa050515af8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9700a70f0a7f428fb066dfa050515af8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34043 } }
I20260620 04:25:25.409466 5898 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9700a70f0a7f428fb066dfa050515af8 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:25:25.410714 5905 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9700a70f0a7f428fb066dfa050515af8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9700a70f0a7f428fb066dfa050515af8. Latest consensus state: current_term: 1 leader_uuid: "9700a70f0a7f428fb066dfa050515af8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9700a70f0a7f428fb066dfa050515af8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34043 } } }
I20260620 04:25:25.410667 5904 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9700a70f0a7f428fb066dfa050515af8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9700a70f0a7f428fb066dfa050515af8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9700a70f0a7f428fb066dfa050515af8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34043 } } }
I20260620 04:25:25.410820 5905 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9700a70f0a7f428fb066dfa050515af8 [sys.catalog]: This master's current role is: LEADER
I20260620 04:25:25.410820 5904 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9700a70f0a7f428fb066dfa050515af8 [sys.catalog]: This master's current role is: LEADER
I20260620 04:25:25.411216 5914 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:25:25.414278 5914 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:25:25.419610 5914 catalog_manager.cc:1347] Generated new cluster ID: ead8406204bb4e1ca411f035edb16a50
I20260620 04:25:25.419680 5914 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:25:25.459991 5914 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:25:25.461262 5914 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:25:25.474068 5914 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9700a70f0a7f428fb066dfa050515af8: Generated new TSK 0
I20260620 04:25:25.474888 5914 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 04:25:25.953052 5901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:25.489831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60268 (local address 127.0.98.62:34043)
0620 04:25:25.489966 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:25.489970 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:25.490200 (+ 230us) server_negotiation.cc:407] Connection header received
0620 04:25:25.490353 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:25.490356 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:25.490411 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:25.490532 (+ 121us) server_negotiation.cc:226] Negotiated authn=JWT
0620 04:25:25.490739 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:25.491511 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:25.492443 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:25.492794 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:25.492890 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:25.493178 (+ 288us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 04:25:25.952661 (+459483us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 04:25:25.952937 (+ 276us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60268: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":51}
W20260620 04:25:25.953126 5901 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60268: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260620 04:25:25.953291 5927 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:25:25.489769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34043 (local address 127.0.0.1:60268)
0620 04:25:25.490112 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:25.490123 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:25.490228 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:25.490533 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:25.490536 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:25.490542 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0620 04:25:25.490644 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:25.490650 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:25.491632 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:25.491636 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:25.492333 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:25.492342 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:25.492888 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:25.492908 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:25.492977 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:25.493001 (+ 24us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 04:25:25.952891 (+459890us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 04:25:25.953074 (+ 183us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:34043: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":97,"threads_started":1}
I20260620 04:25:25.953872 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 5831
2026-06-20T04:25:25Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (760 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (119 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (117 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929496330398-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:25:26 dist-test-slave-nk6d krb5kdc[5939](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:25:26 dist-test-slave-nk6d krb5kdc[5939](info): set up 2 sockets
Jun 20 04:25:26 dist-test-slave-nk6d krb5kdc[5939](info): commencing operation
krb5kdc: starting...
W20260620 04:25:27.140337 392 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929496330398-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929496330398-392-0/krb5kdc/test-user.keytab.
Jun 20 04:25:27 dist-test-slave-nk6d krb5kdc[5939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929527, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:25:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:25:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 04:25:27.252942 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929496330398-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929496330398-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929496330398-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37339
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37339
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:25:27.360754 5952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:25:27.361016 5952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:25:27.361053 5952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:25:27.365060 5952 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:25:27.365123 5952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:25:27.365141 5952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:25:27.365156 5952 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:25:27.365172 5952 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:25:27.370638 5952 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929496330398-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929496330398-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37339
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37339
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5952
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929496330398-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:25:27.371757 5952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:25:27.372815 5952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:25:27.381068 5958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:25:27.381049 5960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:25:27.381131 5957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:25:27.381258 5952 server_base.cc:1034] running on GCE node
I20260620 04:25:27.381980 5952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:25:27.383080 5952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:25:27.384224 5952 hybrid_clock.cc:648] HybridClock initialized: now 1781929527384199 us; error 38 us; skew 500 ppm
Jun 20 04:25:27 dist-test-slave-nk6d krb5kdc[5939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929527, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:25:27.386821 5952 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 04:25:27.387676 5952 webserver.cc:460] Webserver started at http://127.0.98.62:46723/ using document root <none> and password file <none>
I20260620 04:25:27.388278 5952 fs_manager.cc:359] Metadata directory not provided
I20260620 04:25:27.388341 5952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:25:27.388517 5952 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:25:27.390477 5952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929496330398-392-0/minicluster-data/master-0/data/instance:
uuid: "a47db8840ab240afb13d0bf2f596ca93"
format_stamp: "Formatted at 2026-06-20 04:25:27 on dist-test-slave-nk6d"
server_key: "e90311192e2547f2d53e23c94e91c1b3"
server_key_iv: "b2cbc4c4523766ee791f589e25f5cc3e"
server_key_version: "encryptionkey@0"
I20260620 04:25:27.390998 5952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929496330398-392-0/minicluster-data/master-0/wal/instance:
uuid: "a47db8840ab240afb13d0bf2f596ca93"
format_stamp: "Formatted at 2026-06-20 04:25:27 on dist-test-slave-nk6d"
server_key: "e90311192e2547f2d53e23c94e91c1b3"
server_key_iv: "b2cbc4c4523766ee791f589e25f5cc3e"
server_key_version: "encryptionkey@0"
I20260620 04:25:27.394563 5952 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 04:25:27.396817 5967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:27.398095 5952 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:25:27.398198 5952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929496330398-392-0/minicluster-data/master-0/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929496330398-392-0/minicluster-data/master-0/wal
uuid: "a47db8840ab240afb13d0bf2f596ca93"
format_stamp: "Formatted at 2026-06-20 04:25:27 on dist-test-slave-nk6d"
server_key: "e90311192e2547f2d53e23c94e91c1b3"
server_key_iv: "b2cbc4c4523766ee791f589e25f5cc3e"
server_key_version: "encryptionkey@0"
I20260620 04:25:27.398285 5952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929496330398-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929496330398-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929496330398-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:25:27.418210 5952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:25:27.421108 5952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:25:27.421283 5952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:25:27.428449 5952 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37339
I20260620 04:25:27.428689 6019 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37339 every 8 connection(s)
I20260620 04:25:27.429414 5952 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:25:27.432101 6020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:25:27.437354 6020 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a47db8840ab240afb13d0bf2f596ca93: Bootstrap starting.
I20260620 04:25:27.438783 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 5952
I20260620 04:25:27.438902 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929496330398-392-0/minicluster-data/master-0/wal/instance
I20260620 04:25:27.439201 392 external_mini_cluster.cc:1428] Setting key c3293b33040f6dd8ff1409e364bbeb99
I20260620 04:25:27.439761 6020 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a47db8840ab240afb13d0bf2f596ca93: Neither blocks nor log segments found. Creating new log.
I20260620 04:25:27.440644 6020 log.cc:824] T 00000000000000000000000000000000 P a47db8840ab240afb13d0bf2f596ca93: Log is configured to *not* fsync() on all Append() calls
I20260620 04:25:27.442636 6020 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a47db8840ab240afb13d0bf2f596ca93: No bootstrap required, opened a new log
I20260620 04:25:27.446573 6020 raft_consensus.cc:348] T 00000000000000000000000000000000 P a47db8840ab240afb13d0bf2f596ca93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a47db8840ab240afb13d0bf2f596ca93" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37339 } }
I20260620 04:25:27.446879 6020 raft_consensus.cc:374] T 00000000000000000000000000000000 P a47db8840ab240afb13d0bf2f596ca93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:25:27.446928 6020 raft_consensus.cc:729] T 00000000000000000000000000000000 P a47db8840ab240afb13d0bf2f596ca93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a47db8840ab240afb13d0bf2f596ca93, State: Initialized, Role: FOLLOWER
I20260620 04:25:27.447402 6020 consensus_queue.cc:260] T 00000000000000000000000000000000 P a47db8840ab240afb13d0bf2f596ca93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a47db8840ab240afb13d0bf2f596ca93" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37339 } }
I20260620 04:25:27.447530 6020 raft_consensus.cc:388] T 00000000000000000000000000000000 P a47db8840ab240afb13d0bf2f596ca93 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:25:27.447582 6020 raft_consensus.cc:482] T 00000000000000000000000000000000 P a47db8840ab240afb13d0bf2f596ca93 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:25:27.447674 6020 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a47db8840ab240afb13d0bf2f596ca93 [term 0 FOLLOWER]: Advancing to term 1
Jun 20 04:25:27 dist-test-slave-nk6d krb5kdc[5939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929527, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:25:27.448673 6020 raft_consensus.cc:504] T 00000000000000000000000000000000 P a47db8840ab240afb13d0bf2f596ca93 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a47db8840ab240afb13d0bf2f596ca93" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37339 } }
I20260620 04:25:27.449025 6020 leader_election.cc:302] T 00000000000000000000000000000000 P a47db8840ab240afb13d0bf2f596ca93 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a47db8840ab240afb13d0bf2f596ca93; no voters:
I20260620 04:25:27.449383 6020 leader_election.cc:288] T 00000000000000000000000000000000 P a47db8840ab240afb13d0bf2f596ca93 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:25:27.449501 6025 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a47db8840ab240afb13d0bf2f596ca93 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:25:27.449744 6025 raft_consensus.cc:686] T 00000000000000000000000000000000 P a47db8840ab240afb13d0bf2f596ca93 [term 1 LEADER]: Becoming Leader. State: Replica: a47db8840ab240afb13d0bf2f596ca93, State: Running, Role: LEADER
I20260620 04:25:27.450078 6025 consensus_queue.cc:237] T 00000000000000000000000000000000 P a47db8840ab240afb13d0bf2f596ca93 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a47db8840ab240afb13d0bf2f596ca93" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37339 } }
I20260620 04:25:27.450680 6020 sys_catalog.cc:564] T 00000000000000000000000000000000 P a47db8840ab240afb13d0bf2f596ca93 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:25:27.451592 6027 sys_catalog.cc:455] T 00000000000000000000000000000000 P a47db8840ab240afb13d0bf2f596ca93 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a47db8840ab240afb13d0bf2f596ca93. Latest consensus state: current_term: 1 leader_uuid: "a47db8840ab240afb13d0bf2f596ca93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a47db8840ab240afb13d0bf2f596ca93" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37339 } } }
I20260620 04:25:27.451702 6027 sys_catalog.cc:458] T 00000000000000000000000000000000 P a47db8840ab240afb13d0bf2f596ca93 [sys.catalog]: This master's current role is: LEADER
I20260620 04:25:27.452008 6026 sys_catalog.cc:455] T 00000000000000000000000000000000 P a47db8840ab240afb13d0bf2f596ca93 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a47db8840ab240afb13d0bf2f596ca93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a47db8840ab240afb13d0bf2f596ca93" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37339 } } }
I20260620 04:25:27.452091 6026 sys_catalog.cc:458] T 00000000000000000000000000000000 P a47db8840ab240afb13d0bf2f596ca93 [sys.catalog]: This master's current role is: LEADER
I20260620 04:25:27.454293 6023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:27.440243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56272 (local address 127.0.98.62:37339)
0620 04:25:27.440622 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:27.440637 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:27.440671 (+ 34us) server_negotiation.cc:407] Connection header received
0620 04:25:27.441409 (+ 738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:27.441432 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:27.441760 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:27.442037 (+ 277us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:27.442844 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:27.443877 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:27.445032 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:27.445488 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:27.445600 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:27.448416 (+ 2816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:27.448423 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:27.448439 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:25:27.448471 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:27.450400 (+ 1929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:27.450922 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:27.450927 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:27.450933 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:27.451082 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:27.451394 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:27.451398 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:27.451399 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:27.451922 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:27.452722 (+ 800us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:27.452882 (+ 160us) server_negotiation.cc:299] Negotiation successful
0620 04:25:27.453345 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":103,"threads_started":1}
W20260620 04:25:27.455837 6041 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a47db8840ab240afb13d0bf2f596ca93: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 04:25:27.455905 6041 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 04:25:27.455989 6034 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:25:27.456849 6034 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:25:27.461474 6034 catalog_manager.cc:1347] Generated new cluster ID: 7259f39e0c5c47fea0b1274fc10f5ce1
I20260620 04:25:27.461534 6034 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:25:27.471789 6034 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:25:27.473042 6034 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:25:27.480612 6034 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a47db8840ab240afb13d0bf2f596ca93: Generated new TSK 0
I20260620 04:25:27.481420 6034 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929496330398-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929496330398-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 04:25:27.513624 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929496330398-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929496330398-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929496330398-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929496330398-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37339
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929496330398-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:25:27.639539 6046 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:25:27.639822 6046 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:25:27.639890 6046 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:25:27.643924 6046 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:25:27.644014 6046 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:25:27.644100 6046 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 04:25:27.649647 6046 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929496330398-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929496330398-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929496330398-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929496330398-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37339
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6046
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929496330398-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:25:27.650794 6046 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:25:27.651912 6046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:25:27.661247 6054 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:25:27.661257 6052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:25:27.661259 6051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:25:27.661397 6046 server_base.cc:1034] running on GCE node
I20260620 04:25:27.661861 6046 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:25:27.662621 6046 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:25:27.663820 6046 hybrid_clock.cc:648] HybridClock initialized: now 1781929527663791 us; error 46 us; skew 500 ppm
Jun 20 04:25:27 dist-test-slave-nk6d krb5kdc[5939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929527, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:25:27.666538 6046 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 04:25:27.667398 6046 webserver.cc:460] Webserver started at http://127.0.98.1:40735/ using document root <none> and password file <none>
I20260620 04:25:27.667897 6046 fs_manager.cc:359] Metadata directory not provided
I20260620 04:25:27.667951 6046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:25:27.668167 6046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:25:27.670089 6046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929496330398-392-0/minicluster-data/ts-0/data/instance:
uuid: "68971b5fcb1443b187b3d2a6488f4b00"
format_stamp: "Formatted at 2026-06-20 04:25:27 on dist-test-slave-nk6d"
server_key: "89c5ef592a0d1afad093c3e8d7886f10"
server_key_iv: "7efca204a595c1c8f6e63215234b1a2a"
server_key_version: "encryptionkey@0"
I20260620 04:25:27.670565 6046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929496330398-392-0/minicluster-data/ts-0/wal/instance:
uuid: "68971b5fcb1443b187b3d2a6488f4b00"
format_stamp: "Formatted at 2026-06-20 04:25:27 on dist-test-slave-nk6d"
server_key: "89c5ef592a0d1afad093c3e8d7886f10"
server_key_iv: "7efca204a595c1c8f6e63215234b1a2a"
server_key_version: "encryptionkey@0"
I20260620 04:25:27.674098 6046 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 04:25:27.676338 6061 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:27.677510 6046 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:25:27.677644 6046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929496330398-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929496330398-392-0/minicluster-data/ts-0/wal
uuid: "68971b5fcb1443b187b3d2a6488f4b00"
format_stamp: "Formatted at 2026-06-20 04:25:27 on dist-test-slave-nk6d"
server_key: "89c5ef592a0d1afad093c3e8d7886f10"
server_key_iv: "7efca204a595c1c8f6e63215234b1a2a"
server_key_version: "encryptionkey@0"
I20260620 04:25:27.677735 6046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929496330398-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929496330398-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929496330398-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:25:27.697847 6046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:25:27.701006 6046 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:25:27.701198 6046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:25:27.701771 6046 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:25:27.702589 6046 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:25:27.702641 6046 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:27.702692 6046 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:25:27.702720 6046 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:27.711020 6046 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33357
I20260620 04:25:27.711041 6174 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33357 every 8 connection(s)
I20260620 04:25:27.711895 6046 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929496330398-392-0/minicluster-data/ts-0/data/info.pb
I20260620 04:25:27.713382 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 6046
I20260620 04:25:27.713487 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929496330398-392-0/minicluster-data/ts-0/wal/instance
I20260620 04:25:27.713788 392 external_mini_cluster.cc:1428] Setting key a3efc573002730d0fab9e9c2fda2453a
Jun 20 04:25:27 dist-test-slave-nk6d krb5kdc[5939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929527, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 04:25:27.726418 6023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:27.713763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58187 (local address 127.0.98.62:37339)
0620 04:25:27.713911 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:27.713916 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:27.714880 (+ 964us) server_negotiation.cc:407] Connection header received
0620 04:25:27.716330 (+ 1450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:27.716333 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:27.716381 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:27.716461 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:27.718002 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:27.718556 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:27.720251 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:27.720479 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:27.720561 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:27.723173 (+ 2612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:27.723178 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:27.723180 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:25:27.723203 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:27.724487 (+ 1284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:27.724974 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:27.724978 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:27.724979 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:27.725064 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:27.725382 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:27.725386 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:27.725387 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:27.725565 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:27.725658 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:27.726132 (+ 474us) server_negotiation.cc:299] Negotiation successful
0620 04:25:27.726256 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 04:25:27.727263 6177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:27.714193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37339 (local address 127.0.98.1:58187)
0620 04:25:27.714717 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:27.714775 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:27.716132 (+ 1357us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:27.716673 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:27.716682 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:27.717185 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:27.717848 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:27.717868 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:27.718675 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:27.718679 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:27.720114 (+ 1435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:27.720126 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:27.720628 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:27.720632 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:27.720834 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:27.721433 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:27.721455 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:27.723019 (+ 1564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:25:27.724584 (+ 1565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:27.724589 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:27.724604 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:27.724884 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:27.725156 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:27.725159 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:27.725160 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:27.725291 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:27.725667 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:25:27.725674 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:25:27.725912 (+ 238us) client_negotiation.cc:769] Sending connection context
0620 04:25:27.726078 (+ 166us) client_negotiation.cc:240] Negotiation successful
0620 04:25:27.726408 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":130,"threads_started":1}
I20260620 04:25:27.728504 6175 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37339
I20260620 04:25:27.728751 6175 heartbeater.cc:464] Registering TS with master...
I20260620 04:25:27.729205 6175 heartbeater.cc:511] Master 127.0.98.62:37339 requested a full tablet report, sending...
I20260620 04:25:27.730818 5984 ts_manager.cc:194] Registered new tserver with Master: 68971b5fcb1443b187b3d2a6488f4b00 (127.0.98.1:33357)
I20260620 04:25:27.732344 5984 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:58187
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929496330398-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929496330398-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 04:25:27.739131 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929496330398-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929496330398-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929496330398-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929496330398-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37339
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929496330398-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:25:27.844061 6180 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:25:27.844307 6180 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:25:27.844347 6180 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:25:27.849052 6180 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:25:27.849109 6180 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:25:27.849166 6180 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 04:25:27.854319 6180 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929496330398-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929496330398-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929496330398-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929496330398-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37339
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6180
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929496330398-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:25:27.855281 6180 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:25:27.856288 6180 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:25:27.865556 6186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:25:27.865530 6180 server_base.cc:1034] running on GCE node
W20260620 04:25:27.865514 6188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:25:27.865540 6185 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:25:27.866202 6180 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:25:27.866783 6180 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:25:27.868011 6180 hybrid_clock.cc:648] HybridClock initialized: now 1781929527867979 us; error 44 us; skew 500 ppm
Jun 20 04:25:27 dist-test-slave-nk6d krb5kdc[5939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929527, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:25:27.870667 6180 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 04:25:27.871560 6180 webserver.cc:460] Webserver started at http://127.0.98.2:33123/ using document root <none> and password file <none>
I20260620 04:25:27.872121 6180 fs_manager.cc:359] Metadata directory not provided
I20260620 04:25:27.872186 6180 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:25:27.872351 6180 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:25:27.874295 6180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929496330398-392-0/minicluster-data/ts-1/data/instance:
uuid: "27bc97d9e21243b5a15f289af83f2af5"
format_stamp: "Formatted at 2026-06-20 04:25:27 on dist-test-slave-nk6d"
server_key: "b65d06270ed637b9129e52c13d9d5cf3"
server_key_iv: "505edda8bb2a6e544bcec98276adfd17"
server_key_version: "encryptionkey@0"
I20260620 04:25:27.874773 6180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929496330398-392-0/minicluster-data/ts-1/wal/instance:
uuid: "27bc97d9e21243b5a15f289af83f2af5"
format_stamp: "Formatted at 2026-06-20 04:25:27 on dist-test-slave-nk6d"
server_key: "b65d06270ed637b9129e52c13d9d5cf3"
server_key_iv: "505edda8bb2a6e544bcec98276adfd17"
server_key_version: "encryptionkey@0"
I20260620 04:25:27.878268 6180 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 04:25:27.880633 6195 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:27.881842 6180 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:25:27.881935 6180 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929496330398-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929496330398-392-0/minicluster-data/ts-1/wal
uuid: "27bc97d9e21243b5a15f289af83f2af5"
format_stamp: "Formatted at 2026-06-20 04:25:27 on dist-test-slave-nk6d"
server_key: "b65d06270ed637b9129e52c13d9d5cf3"
server_key_iv: "505edda8bb2a6e544bcec98276adfd17"
server_key_version: "encryptionkey@0"
I20260620 04:25:27.882026 6180 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929496330398-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929496330398-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929496330398-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:25:27.898500 6180 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:25:27.901731 6180 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:25:27.901903 6180 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:25:27.902469 6180 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:25:27.903291 6180 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:25:27.903337 6180 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:27.903388 6180 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:25:27.903424 6180 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:27.911911 6180 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36009
I20260620 04:25:27.911995 6308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36009 every 8 connection(s)
I20260620 04:25:27.913035 6180 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929496330398-392-0/minicluster-data/ts-1/data/info.pb
I20260620 04:25:27.914523 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 6180
I20260620 04:25:27.914623 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929496330398-392-0/minicluster-data/ts-1/wal/instance
I20260620 04:25:27.914901 392 external_mini_cluster.cc:1428] Setting key 9c772c0d24fc1d9338b478eb17b776d9
Jun 20 04:25:27 dist-test-slave-nk6d krb5kdc[5939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929527, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260620 04:25:27.926716 6023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:27.914809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35873 (local address 127.0.98.62:37339)
0620 04:25:27.914943 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:27.914947 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:27.915813 (+ 866us) server_negotiation.cc:407] Connection header received
0620 04:25:27.917189 (+ 1376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:27.917193 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:27.917246 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:27.917320 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:27.918479 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:27.919234 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:27.920492 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:27.920829 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:27.920942 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:27.923296 (+ 2354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:27.923300 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:27.923303 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:25:27.923331 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:27.924709 (+ 1378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:27.925180 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:27.925184 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:27.925186 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:27.925257 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:27.925611 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:27.925615 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:27.925617 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:27.925819 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:27.925930 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:27.926412 (+ 482us) server_negotiation.cc:299] Negotiation successful
0620 04:25:27.926532 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 04:25:27.927485 6311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:27.915189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37339 (local address 127.0.98.2:35873)
0620 04:25:27.915661 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:27.915713 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:27.916982 (+ 1269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:27.917502 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:27.917509 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:27.917868 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:27.918344 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:27.918358 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:27.919367 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:27.919370 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:27.920372 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:27.920381 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:27.920963 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:27.920966 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:27.921103 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:27.921648 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:27.921664 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:27.923132 (+ 1468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:25:27.924815 (+ 1683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:27.924821 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:27.924837 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:27.925088 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:27.925335 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:27.925338 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:27.925340 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:27.925486 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:27.925928 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:25:27.925934 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:25:27.926181 (+ 247us) client_negotiation.cc:769] Sending connection context
0620 04:25:27.926338 (+ 157us) client_negotiation.cc:240] Negotiation successful
0620 04:25:27.926605 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":109,"threads_started":1}
I20260620 04:25:27.928609 6309 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37339
I20260620 04:25:27.928841 6309 heartbeater.cc:464] Registering TS with master...
I20260620 04:25:27.929344 6309 heartbeater.cc:511] Master 127.0.98.62:37339 requested a full tablet report, sending...
I20260620 04:25:27.930366 5984 ts_manager.cc:194] Registered new tserver with Master: 27bc97d9e21243b5a15f289af83f2af5 (127.0.98.2:36009)
I20260620 04:25:27.930910 5984 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:35873
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929496330398-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929496330398-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 04:25:27.939268 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929496330398-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929496330398-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929496330398-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929496330398-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37339
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929496330398-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:25:28.047019 6314 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:25:28.047284 6314 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:25:28.047327 6314 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:25:28.051358 6314 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:25:28.051429 6314 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:25:28.051489 6314 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 04:25:28.056806 6314 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929496330398-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929496330398-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929496330398-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929496330398-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37339
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6314
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929496330398-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:25:28.057827 6314 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:25:28.058837 6314 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:25:28.068107 6322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:25:28.068161 6319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:25:28.068154 6314 server_base.cc:1034] running on GCE node
W20260620 04:25:28.068102 6320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:25:28.068717 6314 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:25:28.069394 6314 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:25:28.070571 6314 hybrid_clock.cc:648] HybridClock initialized: now 1781929528070551 us; error 36 us; skew 500 ppm
Jun 20 04:25:28 dist-test-slave-nk6d krb5kdc[5939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929528, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:25:28.073282 6314 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 04:25:28.074092 6314 webserver.cc:460] Webserver started at http://127.0.98.3:37685/ using document root <none> and password file <none>
I20260620 04:25:28.074615 6314 fs_manager.cc:359] Metadata directory not provided
I20260620 04:25:28.074680 6314 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:25:28.074857 6314 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:25:28.076848 6314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929496330398-392-0/minicluster-data/ts-2/data/instance:
uuid: "ce53a969d8184848988a8a509aeb2fa9"
format_stamp: "Formatted at 2026-06-20 04:25:28 on dist-test-slave-nk6d"
server_key: "9681af6b41ebe5bac2a4812a8b34c1bd"
server_key_iv: "bece8a23ddd58d9842ad8b1eccbce0ee"
server_key_version: "encryptionkey@0"
I20260620 04:25:28.077384 6314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929496330398-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ce53a969d8184848988a8a509aeb2fa9"
format_stamp: "Formatted at 2026-06-20 04:25:28 on dist-test-slave-nk6d"
server_key: "9681af6b41ebe5bac2a4812a8b34c1bd"
server_key_iv: "bece8a23ddd58d9842ad8b1eccbce0ee"
server_key_version: "encryptionkey@0"
I20260620 04:25:28.081017 6314 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 04:25:28.083393 6329 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:28.084633 6314 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:25:28.084738 6314 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929496330398-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929496330398-392-0/minicluster-data/ts-2/wal
uuid: "ce53a969d8184848988a8a509aeb2fa9"
format_stamp: "Formatted at 2026-06-20 04:25:28 on dist-test-slave-nk6d"
server_key: "9681af6b41ebe5bac2a4812a8b34c1bd"
server_key_iv: "bece8a23ddd58d9842ad8b1eccbce0ee"
server_key_version: "encryptionkey@0"
I20260620 04:25:28.084826 6314 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929496330398-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929496330398-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929496330398-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:25:28.104091 6314 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:25:28.107329 6314 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:25:28.107482 6314 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:25:28.108038 6314 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:25:28.108878 6314 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:25:28.108927 6314 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:28.108964 6314 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:25:28.108978 6314 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:28.117250 6314 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33055
I20260620 04:25:28.117331 6442 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33055 every 8 connection(s)
I20260620 04:25:28.118172 6314 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929496330398-392-0/minicluster-data/ts-2/data/info.pb
I20260620 04:25:28.125105 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 6314
I20260620 04:25:28.125200 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929496330398-392-0/minicluster-data/ts-2/wal/instance
I20260620 04:25:28.125435 392 external_mini_cluster.cc:1428] Setting key bcab85416bc1cf90e88eab00a11eeb97
Jun 20 04:25:28 dist-test-slave-nk6d krb5kdc[5939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929528, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:25:28.131541 6023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:28.120043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48027 (local address 127.0.98.62:37339)
0620 04:25:28.120196 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:28.120199 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:28.121082 (+ 883us) server_negotiation.cc:407] Connection header received
0620 04:25:28.122460 (+ 1378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:28.122463 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:28.122509 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:28.122588 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:28.123750 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:28.124380 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:28.125594 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:28.125847 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:28.125953 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:28.128460 (+ 2507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:28.128464 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:28.128467 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:25:28.128493 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:28.129679 (+ 1186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:28.130135 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:28.130139 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:28.130141 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:28.130215 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:28.130512 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:28.130515 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:28.130517 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:28.130694 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:28.130778 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:28.131256 (+ 478us) server_negotiation.cc:299] Negotiation successful
0620 04:25:28.131375 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 04:25:28.132380 6445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:28.120418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37339 (local address 127.0.98.3:48027)
0620 04:25:28.120916 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:28.120959 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:28.122256 (+ 1297us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:28.122739 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:28.122745 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:28.123138 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:28.123613 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:28.123627 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:28.124489 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:28.124492 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:28.125443 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:28.125453 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:28.125982 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:28.125985 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:28.126206 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:28.126803 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:28.126820 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:28.128309 (+ 1489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:25:28.129782 (+ 1473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:28.129788 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:28.129804 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:28.130046 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:28.130293 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:28.130295 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:28.130298 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:28.130430 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:28.130788 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:25:28.130795 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:25:28.131036 (+ 241us) client_negotiation.cc:769] Sending connection context
0620 04:25:28.131204 (+ 168us) client_negotiation.cc:240] Negotiation successful
0620 04:25:28.131521 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":121,"threads_started":1}
I20260620 04:25:28.133424 6443 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37339
I20260620 04:25:28.133651 6443 heartbeater.cc:464] Registering TS with master...
I20260620 04:25:28.134148 6443 heartbeater.cc:511] Master 127.0.98.62:37339 requested a full tablet report, sending...
I20260620 04:25:28.135114 5984 ts_manager.cc:194] Registered new tserver with Master: ce53a969d8184848988a8a509aeb2fa9 (127.0.98.3:33055)
I20260620 04:25:28.135691 5984 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:48027
I20260620 04:25:28.138953 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929496330398-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929496330398-392-0/krb5kdc/test-admin.keytab.
Jun 20 04:25:28 dist-test-slave-nk6d krb5kdc[5939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:25:28.153414 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260620 04:25:28.156121 6450 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:25:28.154752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35800 (local address 127.0.98.1:33357)
0620 04:25:28.155069 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:28.155082 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:28.155106 (+ 24us) server_negotiation.cc:407] Connection header received
0620 04:25:28.155168 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:28.155175 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:28.155362 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:28.155492 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:28.155934 (+ 442us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35800: BlockingRecv error: recv got EOF from 127.0.0.1:35800 (error 108)
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":102,"threads_started":1}
W20260620 04:25:28.156419 392 init.cc:244] Error while opening credential cache '/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929496330398-392-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929496330398-392-0/krb5kdc/krb5cc)
Jun 20 04:25:28 dist-test-slave-nk6d krb5kdc[5939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:25:28.157409 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 04:25:28 dist-test-slave-nk6d krb5kdc[5939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:25:28.158325 392 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 04:25:28 dist-test-slave-nk6d krb5kdc[5939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 04:25:28.167507 6450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:28.159251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35816 (local address 127.0.98.1:33357)
0620 04:25:28.159346 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:28.159350 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:28.159362 (+ 12us) server_negotiation.cc:407] Connection header received
0620 04:25:28.159512 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:28.159514 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:28.159558 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:28.159635 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:28.160240 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:28.160999 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:28.161779 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:28.162175 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:28.162256 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:28.164521 (+ 2265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:28.164530 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:28.164543 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:25:28.164566 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:28.166011 (+ 1445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:28.166388 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:28.166392 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:28.166393 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:28.166460 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:28.166671 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:28.166673 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:28.166674 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:28.166910 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:28.167067 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:28.167251 (+ 184us) server_negotiation.cc:299] Negotiation successful
0620 04:25:28.167365 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
W20260620 04:25:28.171092 6450 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:25:28.170260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35832 (local address 127.0.98.1:33357)
0620 04:25:28.170381 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:28.170385 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:28.170471 (+ 86us) server_negotiation.cc:407] Connection header received
0620 04:25:28.170547 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:28.170549 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:28.170583 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:28.170641 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:28.170995 (+ 354us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35832: BlockingRecv error: recv got EOF from 127.0.0.1:35832 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260620 04:25:28.171329 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929496330398-392-0/krb5kdc/krb5cc' of type FILE
Jun 20 04:25:28 dist-test-slave-nk6d krb5kdc[5939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:25:28.172322 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 04:25:28 dist-test-slave-nk6d krb5kdc[5939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:25:28.173167 392 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 04:25:28 dist-test-slave-nk6d krb5kdc[5939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 04:25:28.181237 6450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:28.173917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35838 (local address 127.0.98.1:33357)
0620 04:25:28.174014 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:28.174017 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:28.174130 (+ 113us) server_negotiation.cc:407] Connection header received
0620 04:25:28.174224 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:28.174226 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:28.174261 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:28.174314 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:28.174718 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:28.175242 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:28.176019 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:28.176264 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:28.176338 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:28.178507 (+ 2169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:28.178511 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:28.178514 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:25:28.178539 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:28.179758 (+ 1219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:28.180212 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:28.180214 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:28.180215 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:28.180283 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:28.180541 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:28.180543 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:28.180544 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:28.180692 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:28.180779 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:28.180953 (+ 174us) server_negotiation.cc:299] Negotiation successful
0620 04:25:28.181059 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
W20260620 04:25:28.183732 6450 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:25:28.182843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35852 (local address 127.0.98.1:33357)
0620 04:25:28.182944 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:28.182947 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:28.183065 (+ 118us) server_negotiation.cc:407] Connection header received
0620 04:25:28.183193 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:28.183196 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:28.183230 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:28.183292 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:28.183651 (+ 359us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35852: BlockingRecv error: recv got EOF from 127.0.0.1:35852 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260620 04:25:28.183847 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929496330398-392-0/krb5kdc/krb5cc' of type FILE
Jun 20 04:25:28 dist-test-slave-nk6d krb5kdc[5939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:25:28.184808 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 04:25:28 dist-test-slave-nk6d krb5kdc[5939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:25:28.185587 392 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 04:25:28 dist-test-slave-nk6d krb5kdc[5939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 04:25:28.192924 6450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:28.186337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35854 (local address 127.0.98.1:33357)
0620 04:25:28.186453 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:28.186456 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:28.186509 (+ 53us) server_negotiation.cc:407] Connection header received
0620 04:25:28.186593 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:28.186595 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:28.186626 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:28.186688 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:28.187096 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:28.187579 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:28.188476 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:28.188683 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:28.188752 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:28.190600 (+ 1848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:28.190603 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:28.190605 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:25:28.190625 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:28.191629 (+ 1004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:28.191947 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:28.191949 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:28.191950 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:28.192045 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:28.192281 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:28.192284 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:28.192286 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:28.192449 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:28.192555 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:28.192691 (+ 136us) server_negotiation.cc:299] Negotiation successful
0620 04:25:28.192778 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 04:25:28.193895 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 6046
I20260620 04:25:28.198645 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 6180
I20260620 04:25:28.204516 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 6314
I20260620 04:25:28.209090 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 5952
2026-06-20T04:25:28Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2020 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:25:28 dist-test-slave-nk6d krb5kdc[6463](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:25:28 dist-test-slave-nk6d krb5kdc[6463](info): set up 2 sockets
Jun 20 04:25:28 dist-test-slave-nk6d krb5kdc[6463](info): commencing operation
krb5kdc: starting...
W20260620 04:25:29.164077 392 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/krb5kdc/test-user.keytab.
Jun 20 04:25:29 dist-test-slave-nk6d krb5kdc[6463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929529, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:25:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:25:29Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260620 04:25:29.275367 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37461
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37461
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:25:29.381906 6476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:25:29.382184 6476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:25:29.382241 6476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:25:29.386193 6476 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:25:29.386246 6476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:25:29.386273 6476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:25:29.386293 6476 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:25:29.386314 6476 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:25:29.391494 6476 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37461
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37461
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6476
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:25:29.392532 6476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:25:29.393518 6476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:25:29.401561 6484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:25:29.401687 6476 server_base.cc:1034] running on GCE node
W20260620 04:25:29.401516 6481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:25:29.401521 6482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:25:29.402395 6476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:25:29.403458 6476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:25:29.404608 6476 hybrid_clock.cc:648] HybridClock initialized: now 1781929529404591 us; error 29 us; skew 500 ppm
Jun 20 04:25:29 dist-test-slave-nk6d krb5kdc[6463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929529, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:25:29.407263 6476 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260620 04:25:29.408131 6476 webserver.cc:460] Webserver started at http://127.0.98.62:35289/ using document root <none> and password file <none>
I20260620 04:25:29.408645 6476 fs_manager.cc:359] Metadata directory not provided
I20260620 04:25:29.408708 6476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:25:29.408917 6476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:25:29.410954 6476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/master-0/data/instance:
uuid: "961f73cd6ac541f79ace3f8d78ff255b"
format_stamp: "Formatted at 2026-06-20 04:25:29 on dist-test-slave-nk6d"
server_key: "f69235b24cebb09899e68e070748504f"
server_key_iv: "02059cce98bd06b8a675b3070ee7af98"
server_key_version: "encryptionkey@0"
I20260620 04:25:29.411477 6476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/master-0/wal/instance:
uuid: "961f73cd6ac541f79ace3f8d78ff255b"
format_stamp: "Formatted at 2026-06-20 04:25:29 on dist-test-slave-nk6d"
server_key: "f69235b24cebb09899e68e070748504f"
server_key_iv: "02059cce98bd06b8a675b3070ee7af98"
server_key_version: "encryptionkey@0"
I20260620 04:25:29.415138 6476 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 04:25:29.417527 6491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:29.418766 6476 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 04:25:29.418900 6476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/master-0/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/master-0/wal
uuid: "961f73cd6ac541f79ace3f8d78ff255b"
format_stamp: "Formatted at 2026-06-20 04:25:29 on dist-test-slave-nk6d"
server_key: "f69235b24cebb09899e68e070748504f"
server_key_iv: "02059cce98bd06b8a675b3070ee7af98"
server_key_version: "encryptionkey@0"
I20260620 04:25:29.418995 6476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:25:29.435096 6476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:25:29.438338 6476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:25:29.438521 6476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:25:29.445382 6476 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37461
I20260620 04:25:29.445641 6543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37461 every 8 connection(s)
I20260620 04:25:29.446388 6476 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:25:29.449234 6544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:25:29.451038 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 6476
I20260620 04:25:29.451153 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/master-0/wal/instance
I20260620 04:25:29.451421 392 external_mini_cluster.cc:1428] Setting key dcb81f9866c19ab2b3cca42d2d627a65
I20260620 04:25:29.455448 6544 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 961f73cd6ac541f79ace3f8d78ff255b: Bootstrap starting.
I20260620 04:25:29.457602 6544 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 961f73cd6ac541f79ace3f8d78ff255b: Neither blocks nor log segments found. Creating new log.
I20260620 04:25:29.458295 6544 log.cc:824] T 00000000000000000000000000000000 P 961f73cd6ac541f79ace3f8d78ff255b: Log is configured to *not* fsync() on all Append() calls
Jun 20 04:25:29 dist-test-slave-nk6d krb5kdc[6463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929529, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 04:25:29.460271 6544 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 961f73cd6ac541f79ace3f8d78ff255b: No bootstrap required, opened a new log
I20260620 04:25:29.463604 6544 raft_consensus.cc:348] T 00000000000000000000000000000000 P 961f73cd6ac541f79ace3f8d78ff255b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "961f73cd6ac541f79ace3f8d78ff255b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37461 } }
I20260620 04:25:29.463826 6544 raft_consensus.cc:374] T 00000000000000000000000000000000 P 961f73cd6ac541f79ace3f8d78ff255b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:25:29.463861 6544 raft_consensus.cc:729] T 00000000000000000000000000000000 P 961f73cd6ac541f79ace3f8d78ff255b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 961f73cd6ac541f79ace3f8d78ff255b, State: Initialized, Role: FOLLOWER
I20260620 04:25:29.464301 6547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:29.452803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42866 (local address 127.0.98.62:37461)
0620 04:25:29.453216 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:29.453230 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:29.453265 (+ 35us) server_negotiation.cc:407] Connection header received
0620 04:25:29.453907 (+ 642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:29.453923 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:29.454244 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:29.454554 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:29.455329 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:29.456343 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:29.457136 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:29.457593 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:29.457731 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:29.460151 (+ 2420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:29.460156 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:29.460169 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:25:29.460195 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:29.461704 (+ 1509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:29.462085 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:29.462090 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:29.462095 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:29.462196 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:29.462442 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:29.462445 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:29.462447 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:29.462823 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:29.463003 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:29.463260 (+ 257us) server_negotiation.cc:299] Negotiation successful
0620 04:25:29.463522 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":118,"threads_started":1}
I20260620 04:25:29.464320 6544 consensus_queue.cc:260] T 00000000000000000000000000000000 P 961f73cd6ac541f79ace3f8d78ff255b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "961f73cd6ac541f79ace3f8d78ff255b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37461 } }
I20260620 04:25:29.464448 6544 raft_consensus.cc:388] T 00000000000000000000000000000000 P 961f73cd6ac541f79ace3f8d78ff255b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:25:29.464509 6544 raft_consensus.cc:482] T 00000000000000000000000000000000 P 961f73cd6ac541f79ace3f8d78ff255b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:25:29.464598 6544 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 961f73cd6ac541f79ace3f8d78ff255b [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:25:29.465626 6544 raft_consensus.cc:504] T 00000000000000000000000000000000 P 961f73cd6ac541f79ace3f8d78ff255b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "961f73cd6ac541f79ace3f8d78ff255b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37461 } }
I20260620 04:25:29.465948 6544 leader_election.cc:302] T 00000000000000000000000000000000 P 961f73cd6ac541f79ace3f8d78ff255b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 961f73cd6ac541f79ace3f8d78ff255b; no voters:
I20260620 04:25:29.466218 6544 leader_election.cc:288] T 00000000000000000000000000000000 P 961f73cd6ac541f79ace3f8d78ff255b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:25:29.466321 6549 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 961f73cd6ac541f79ace3f8d78ff255b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:25:29.466513 6549 raft_consensus.cc:686] T 00000000000000000000000000000000 P 961f73cd6ac541f79ace3f8d78ff255b [term 1 LEADER]: Becoming Leader. State: Replica: 961f73cd6ac541f79ace3f8d78ff255b, State: Running, Role: LEADER
I20260620 04:25:29.466888 6549 consensus_queue.cc:237] T 00000000000000000000000000000000 P 961f73cd6ac541f79ace3f8d78ff255b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "961f73cd6ac541f79ace3f8d78ff255b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37461 } }
I20260620 04:25:29.467327 6544 sys_catalog.cc:564] T 00000000000000000000000000000000 P 961f73cd6ac541f79ace3f8d78ff255b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:25:29.467931 6551 sys_catalog.cc:455] T 00000000000000000000000000000000 P 961f73cd6ac541f79ace3f8d78ff255b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 961f73cd6ac541f79ace3f8d78ff255b. Latest consensus state: current_term: 1 leader_uuid: "961f73cd6ac541f79ace3f8d78ff255b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "961f73cd6ac541f79ace3f8d78ff255b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37461 } } }
I20260620 04:25:29.468144 6551 sys_catalog.cc:458] T 00000000000000000000000000000000 P 961f73cd6ac541f79ace3f8d78ff255b [sys.catalog]: This master's current role is: LEADER
I20260620 04:25:29.468343 6550 sys_catalog.cc:455] T 00000000000000000000000000000000 P 961f73cd6ac541f79ace3f8d78ff255b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "961f73cd6ac541f79ace3f8d78ff255b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "961f73cd6ac541f79ace3f8d78ff255b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37461 } } }
I20260620 04:25:29.468423 6550 sys_catalog.cc:458] T 00000000000000000000000000000000 P 961f73cd6ac541f79ace3f8d78ff255b [sys.catalog]: This master's current role is: LEADER
I20260620 04:25:29.468597 6558 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:25:29.471622 6558 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:25:29.477375 6558 catalog_manager.cc:1347] Generated new cluster ID: 65e72dfb217a4b24b6c29af381fead6f
I20260620 04:25:29.477432 6558 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:25:29.484297 6558 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:25:29.485188 6558 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:25:29.496059 6558 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 961f73cd6ac541f79ace3f8d78ff255b: Generated new TSK 0
I20260620 04:25:29.496949 6558 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260620 04:25:29.534250 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37461
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:25:29.663208 6570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:25:29.663475 6570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:25:29.663515 6570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:25:29.667425 6570 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:25:29.667507 6570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:25:29.667582 6570 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 04:25:29.672788 6570 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37461
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6570
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:25:29.673815 6570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:25:29.674811 6570 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:25:29.684154 6575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:25:29.684178 6570 server_base.cc:1034] running on GCE node
W20260620 04:25:29.684149 6578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:25:29.684154 6576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:25:29.684793 6570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:25:29.685403 6570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:25:29.686553 6570 hybrid_clock.cc:648] HybridClock initialized: now 1781929529686535 us; error 31 us; skew 500 ppm
Jun 20 04:25:29 dist-test-slave-nk6d krb5kdc[6463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929529, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:25:29.689112 6570 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260620 04:25:29.689939 6570 webserver.cc:460] Webserver started at http://127.0.98.1:33081/ using document root <none> and password file <none>
I20260620 04:25:29.690451 6570 fs_manager.cc:359] Metadata directory not provided
I20260620 04:25:29.690513 6570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:25:29.690699 6570 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:25:29.692673 6570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/ts-0/data/instance:
uuid: "449f41f8c68044e29bb5d65d0f0652ef"
format_stamp: "Formatted at 2026-06-20 04:25:29 on dist-test-slave-nk6d"
server_key: "bcb2a96783a4c927e6532ff891a493fe"
server_key_iv: "4399317d5e3c1fafa96eb38a07b31998"
server_key_version: "encryptionkey@0"
I20260620 04:25:29.693178 6570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/ts-0/wal/instance:
uuid: "449f41f8c68044e29bb5d65d0f0652ef"
format_stamp: "Formatted at 2026-06-20 04:25:29 on dist-test-slave-nk6d"
server_key: "bcb2a96783a4c927e6532ff891a493fe"
server_key_iv: "4399317d5e3c1fafa96eb38a07b31998"
server_key_version: "encryptionkey@0"
I20260620 04:25:29.696594 6570 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 04:25:29.698774 6585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:29.699903 6570 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:25:29.700022 6570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/ts-0/wal
uuid: "449f41f8c68044e29bb5d65d0f0652ef"
format_stamp: "Formatted at 2026-06-20 04:25:29 on dist-test-slave-nk6d"
server_key: "bcb2a96783a4c927e6532ff891a493fe"
server_key_iv: "4399317d5e3c1fafa96eb38a07b31998"
server_key_version: "encryptionkey@0"
I20260620 04:25:29.700109 6570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:25:29.712620 6570 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:25:29.715255 6570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:25:29.715423 6570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:25:29.716004 6570 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:25:29.716770 6570 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:25:29.716823 6570 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:29.716874 6570 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:25:29.716902 6570 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:29.725107 6570 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42601
I20260620 04:25:29.725126 6698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42601 every 8 connection(s)
I20260620 04:25:29.725955 6570 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/ts-0/data/info.pb
I20260620 04:25:29.730971 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 6570
I20260620 04:25:29.731084 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/ts-0/wal/instance
I20260620 04:25:29.731338 392 external_mini_cluster.cc:1428] Setting key 9698834da98ee30dcc7905d2bb8eb9d4
Jun 20 04:25:29 dist-test-slave-nk6d krb5kdc[6463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929529, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 04:25:29.740140 6547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:29.727755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59173 (local address 127.0.98.62:37461)
0620 04:25:29.727898 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:29.727902 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:29.728895 (+ 993us) server_negotiation.cc:407] Connection header received
0620 04:25:29.730482 (+ 1587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:29.730485 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:29.730532 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:29.730603 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:29.731912 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:29.732525 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:29.733699 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:29.734019 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:29.734105 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:29.736805 (+ 2700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:29.736809 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:29.736811 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:25:29.736833 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:29.738139 (+ 1306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:29.738641 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:29.738645 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:29.738646 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:29.738724 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:29.739029 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:29.739033 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:29.739035 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:29.739205 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:29.739295 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:29.739811 (+ 516us) server_negotiation.cc:299] Negotiation successful
0620 04:25:29.739942 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 04:25:29.740999 6701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:29.728222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37461 (local address 127.0.98.1:59173)
0620 04:25:29.728710 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:29.728785 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:29.730273 (+ 1488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:29.730816 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:29.730823 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:29.731271 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:29.731784 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:29.731798 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:29.732646 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:29.732649 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:29.733573 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:29.733583 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:29.734170 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:29.734174 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:29.734380 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:29.734982 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:29.735003 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:29.736633 (+ 1630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:25:29.738269 (+ 1636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:29.738275 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:29.738292 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:29.738550 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:29.738813 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:29.738816 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:29.738818 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:29.738946 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:29.739304 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:25:29.739311 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:25:29.739553 (+ 242us) client_negotiation.cc:769] Sending connection context
0620 04:25:29.739767 (+ 214us) client_negotiation.cc:240] Negotiation successful
0620 04:25:29.740120 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":117,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
I20260620 04:25:29.742272 6699 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37461
I20260620 04:25:29.742549 6699 heartbeater.cc:464] Registering TS with master...
I20260620 04:25:29.743124 6699 heartbeater.cc:511] Master 127.0.98.62:37461 requested a full tablet report, sending...
I20260620 04:25:29.744707 6508 ts_manager.cc:194] Registered new tserver with Master: 449f41f8c68044e29bb5d65d0f0652ef (127.0.98.1:42601)
I20260620 04:25:29.745846 6508 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:59173
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260620 04:25:29.754899 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37461
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:25:29.862144 6704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:25:29.862398 6704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:25:29.862444 6704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:25:29.866396 6704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:25:29.866447 6704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:25:29.866506 6704 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 04:25:29.871729 6704 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37461
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6704
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:25:29.872754 6704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:25:29.873740 6704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:25:29.882826 6709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:25:29.882826 6710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:25:29.882835 6712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:25:29.883070 6704 server_base.cc:1034] running on GCE node
I20260620 04:25:29.883526 6704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:25:29.884166 6704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:25:29.885326 6704 hybrid_clock.cc:648] HybridClock initialized: now 1781929529885301 us; error 37 us; skew 500 ppm
Jun 20 04:25:29 dist-test-slave-nk6d krb5kdc[6463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929529, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:25:29.888067 6704 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260620 04:25:29.888923 6704 webserver.cc:460] Webserver started at http://127.0.98.2:43693/ using document root <none> and password file <none>
I20260620 04:25:29.889443 6704 fs_manager.cc:359] Metadata directory not provided
I20260620 04:25:29.889506 6704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:25:29.889708 6704 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:25:29.891635 6704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/ts-1/data/instance:
uuid: "bde7e8cb28e54e59b808c1bb8439dbf1"
format_stamp: "Formatted at 2026-06-20 04:25:29 on dist-test-slave-nk6d"
server_key: "591dc49cdd34f572835a322fdf0c56c2"
server_key_iv: "294da74d391808e62783c48e6aa216f3"
server_key_version: "encryptionkey@0"
I20260620 04:25:29.892179 6704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/ts-1/wal/instance:
uuid: "bde7e8cb28e54e59b808c1bb8439dbf1"
format_stamp: "Formatted at 2026-06-20 04:25:29 on dist-test-slave-nk6d"
server_key: "591dc49cdd34f572835a322fdf0c56c2"
server_key_iv: "294da74d391808e62783c48e6aa216f3"
server_key_version: "encryptionkey@0"
I20260620 04:25:29.895737 6704 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 04:25:29.898079 6719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:29.899251 6704 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:25:29.899349 6704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/ts-1/wal
uuid: "bde7e8cb28e54e59b808c1bb8439dbf1"
format_stamp: "Formatted at 2026-06-20 04:25:29 on dist-test-slave-nk6d"
server_key: "591dc49cdd34f572835a322fdf0c56c2"
server_key_iv: "294da74d391808e62783c48e6aa216f3"
server_key_version: "encryptionkey@0"
I20260620 04:25:29.899436 6704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:25:29.914295 6704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:25:29.917137 6704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:25:29.917338 6704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:25:29.917896 6704 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:25:29.918687 6704 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:25:29.918730 6704 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:29.918784 6704 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:25:29.918813 6704 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:29.927220 6704 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38783
I20260620 04:25:29.927328 6832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38783 every 8 connection(s)
I20260620 04:25:29.928128 6704 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/ts-1/data/info.pb
I20260620 04:25:29.930370 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 6704
I20260620 04:25:29.930474 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/ts-1/wal/instance
I20260620 04:25:29.930796 392 external_mini_cluster.cc:1428] Setting key 7337eeb6f71edf58a9701805f5267ce8
Jun 20 04:25:29 dist-test-slave-nk6d krb5kdc[6463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929529, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
I20260620 04:25:29.942668 6547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:29.929836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35971 (local address 127.0.98.62:37461)
0620 04:25:29.929976 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:29.929980 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:29.931005 (+ 1025us) server_negotiation.cc:407] Connection header received
0620 04:25:29.932531 (+ 1526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:29.932535 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:29.932591 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:29.932676 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:29.933828 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:29.934407 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:29.936045 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:29.936278 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:29.936363 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:29.939146 (+ 2783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:29.939149 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:29.939151 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:25:29.939174 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:29.940668 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:29.941268 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:29.941271 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:29.941272 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:29.941349 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:29.941631 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:29.941633 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:29.941635 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:29.941816 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:29.941905 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:29.942406 (+ 501us) server_negotiation.cc:299] Negotiation successful
0620 04:25:29.942515 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 04:25:29.943645 6835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:29.930352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37461 (local address 127.0.98.2:35971)
0620 04:25:29.930841 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:29.930901 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:29.932314 (+ 1413us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:29.932846 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:29.932853 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:29.933212 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:29.933683 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:29.933698 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:29.934537 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:29.934540 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:29.935891 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:29.935904 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:29.936430 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:29.936435 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:29.936634 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:29.937226 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:29.937250 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:29.938972 (+ 1722us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:25:29.940791 (+ 1819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:29.940796 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:29.940807 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:29.941131 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:29.941424 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:29.941426 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:29.941427 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:29.941551 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:29.941930 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:25:29.941935 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:25:29.942165 (+ 230us) client_negotiation.cc:769] Sending connection context
0620 04:25:29.942348 (+ 183us) client_negotiation.cc:240] Negotiation successful
0620 04:25:29.942661 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":131,"threads_started":1}
I20260620 04:25:29.944911 6833 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37461
I20260620 04:25:29.945143 6833 heartbeater.cc:464] Registering TS with master...
I20260620 04:25:29.945643 6833 heartbeater.cc:511] Master 127.0.98.62:37461 requested a full tablet report, sending...
I20260620 04:25:29.946658 6508 ts_manager.cc:194] Registered new tserver with Master: bde7e8cb28e54e59b808c1bb8439dbf1 (127.0.98.2:38783)
I20260620 04:25:29.947218 6508 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:35971
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260620 04:25:29.955937 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37461
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:25:30.062536 6838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:25:30.062809 6838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:25:30.062870 6838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:25:30.066815 6838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:25:30.066881 6838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:25:30.066982 6838 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 04:25:30.072367 6838 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37461
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6838
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:25:30.073444 6838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:25:30.074447 6838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:25:30.083912 6846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:25:30.083904 6844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:25:30.083945 6843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:25:30.084117 6838 server_base.cc:1034] running on GCE node
I20260620 04:25:30.084527 6838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:25:30.085232 6838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:25:30.086417 6838 hybrid_clock.cc:648] HybridClock initialized: now 1781929530086396 us; error 33 us; skew 500 ppm
Jun 20 04:25:30 dist-test-slave-nk6d krb5kdc[6463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929530, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:25:30.089082 6838 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260620 04:25:30.089902 6838 webserver.cc:460] Webserver started at http://127.0.98.3:34813/ using document root <none> and password file <none>
I20260620 04:25:30.090418 6838 fs_manager.cc:359] Metadata directory not provided
I20260620 04:25:30.090480 6838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:25:30.090656 6838 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:25:30.092657 6838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/ts-2/data/instance:
uuid: "66d7beaf7d8a415f8e5e1b457f1ec504"
format_stamp: "Formatted at 2026-06-20 04:25:30 on dist-test-slave-nk6d"
server_key: "50339e8927dba87a693b0fe89e778b0b"
server_key_iv: "a1ecc2c08068a7f064c9b09db8aacc60"
server_key_version: "encryptionkey@0"
I20260620 04:25:30.093168 6838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/ts-2/wal/instance:
uuid: "66d7beaf7d8a415f8e5e1b457f1ec504"
format_stamp: "Formatted at 2026-06-20 04:25:30 on dist-test-slave-nk6d"
server_key: "50339e8927dba87a693b0fe89e778b0b"
server_key_iv: "a1ecc2c08068a7f064c9b09db8aacc60"
server_key_version: "encryptionkey@0"
I20260620 04:25:30.096752 6838 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 04:25:30.099184 6853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:30.100515 6838 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:25:30.100636 6838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/ts-2/wal
uuid: "66d7beaf7d8a415f8e5e1b457f1ec504"
format_stamp: "Formatted at 2026-06-20 04:25:30 on dist-test-slave-nk6d"
server_key: "50339e8927dba87a693b0fe89e778b0b"
server_key_iv: "a1ecc2c08068a7f064c9b09db8aacc60"
server_key_version: "encryptionkey@0"
I20260620 04:25:30.100734 6838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:25:30.119558 6838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:25:30.122961 6838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:25:30.123150 6838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:25:30.123801 6838 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:25:30.124657 6838 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:25:30.124704 6838 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:30.124758 6838 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:25:30.124784 6838 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:30.133066 6838 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33941
I20260620 04:25:30.133112 6966 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33941 every 8 connection(s)
I20260620 04:25:30.134007 6838 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/ts-2/data/info.pb
I20260620 04:25:30.142004 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 6838
I20260620 04:25:30.142115 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/ts-2/wal/instance
I20260620 04:25:30.142354 392 external_mini_cluster.cc:1428] Setting key 7a19b4a30df18250431125c2b45da121
Jun 20 04:25:30 dist-test-slave-nk6d krb5kdc[6463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929530, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 04:25:30.147261 6547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:30.135925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:32979 (local address 127.0.98.62:37461)
0620 04:25:30.136089 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:30.136094 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:30.136953 (+ 859us) server_negotiation.cc:407] Connection header received
0620 04:25:30.138195 (+ 1242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:30.138198 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:30.138244 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:30.138320 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:30.139429 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:30.139984 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:30.141062 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:30.141268 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:30.141334 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:30.143940 (+ 2606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:30.143943 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:30.143945 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:25:30.143998 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:30.145421 (+ 1423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:30.145906 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:30.145909 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:30.145910 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:30.145992 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:30.146252 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:30.146254 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:30.146255 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:30.146428 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:30.146501 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:30.147047 (+ 546us) server_negotiation.cc:299] Negotiation successful
0620 04:25:30.147144 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 04:25:30.148383 6969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:30.136355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37461 (local address 127.0.98.3:32979)
0620 04:25:30.136804 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:30.136848 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:30.138010 (+ 1162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:30.138493 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:30.138500 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:30.138844 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:30.139306 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:30.139318 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:30.140075 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:30.140077 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:30.140965 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:30.140974 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:30.141366 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:30.141369 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:30.141526 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:30.142132 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:30.142153 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:30.143786 (+ 1633us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:25:30.145523 (+ 1737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:30.145527 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:30.145541 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:30.145790 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:30.146062 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:30.146064 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:30.146065 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:30.146190 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:30.146549 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:25:30.146555 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:25:30.146825 (+ 270us) client_negotiation.cc:769] Sending connection context
0620 04:25:30.147022 (+ 197us) client_negotiation.cc:240] Negotiation successful
0620 04:25:30.147339 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":112,"threads_started":1}
I20260620 04:25:30.149466 6967 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37461
I20260620 04:25:30.149672 6967 heartbeater.cc:464] Registering TS with master...
I20260620 04:25:30.150098 6967 heartbeater.cc:511] Master 127.0.98.62:37461 requested a full tablet report, sending...
I20260620 04:25:30.151031 6508 ts_manager.cc:194] Registered new tserver with Master: 66d7beaf7d8a415f8e5e1b457f1ec504 (127.0.98.3:33941)
I20260620 04:25:30.151547 6508 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:32979
I20260620 04:25:30.156188 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260620 04:25:30.178656 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:34531
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37461,127.0.98.61:34531
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:25:30.289125 6972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:25:30.289402 6972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:25:30.289456 6972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:25:30.293437 6972 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:25:30.293493 6972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:25:30.293519 6972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:25:30.293540 6972 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:25:30.293568 6972 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:25:30.298902 6972 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37461,127.0.98.61:34531
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:34531
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6972
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
W20260620 04:25:30.299130 6972 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:37461,127.0.98.61:34531'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260620 04:25:30.300016 6972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:25:30.300995 6972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:25:30.309237 6980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:25:30.309424 6972 server_base.cc:1034] running on GCE node
W20260620 04:25:30.309229 6977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:25:30.309229 6978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:25:30.310160 6972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:25:30.311250 6972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:25:30.312434 6972 hybrid_clock.cc:648] HybridClock initialized: now 1781929530312416 us; error 34 us; skew 500 ppm
Jun 20 04:25:30 dist-test-slave-nk6d krb5kdc[6463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929530, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:25:30.314980 6972 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260620 04:25:30.315840 6972 webserver.cc:460] Webserver started at http://127.0.98.61:46727/ using document root <none> and password file <none>
I20260620 04:25:30.316385 6972 fs_manager.cc:359] Metadata directory not provided
I20260620 04:25:30.316442 6972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:25:30.316604 6972 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:25:30.318526 6972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/master-1/data/instance:
uuid: "1adca5af9a0644fcb8f274cbc0ae7292"
format_stamp: "Formatted at 2026-06-20 04:25:30 on dist-test-slave-nk6d"
server_key: "f40d2cf75d6a7763f4f86bd99f81129a"
server_key_iv: "5e11fd9ece9550cf1eacf0985bc5e9ea"
server_key_version: "encryptionkey@0"
I20260620 04:25:30.319013 6972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/master-1/wal/instance:
uuid: "1adca5af9a0644fcb8f274cbc0ae7292"
format_stamp: "Formatted at 2026-06-20 04:25:30 on dist-test-slave-nk6d"
server_key: "f40d2cf75d6a7763f4f86bd99f81129a"
server_key_iv: "5e11fd9ece9550cf1eacf0985bc5e9ea"
server_key_version: "encryptionkey@0"
I20260620 04:25:30.322546 6972 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260620 04:25:30.325009 6987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:30.326303 6972 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:25:30.326422 6972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/master-1/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/master-1/wal
uuid: "1adca5af9a0644fcb8f274cbc0ae7292"
format_stamp: "Formatted at 2026-06-20 04:25:30 on dist-test-slave-nk6d"
server_key: "f40d2cf75d6a7763f4f86bd99f81129a"
server_key_iv: "5e11fd9ece9550cf1eacf0985bc5e9ea"
server_key_version: "encryptionkey@0"
I20260620 04:25:30.326532 6972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:25:30.351495 6972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:25:30.357991 6972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:25:30.358176 6972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:25:30.364636 6972 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:34531
I20260620 04:25:30.364902 7039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:34531 every 8 connection(s)
I20260620 04:25:30.365581 6972 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/master-1/data/info.pb
I20260620 04:25:30.368552 7040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:25:30.372368 7040 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37461 } has no permanent_uuid. Determining permanent_uuid...
I20260620 04:25:30.374787 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 6972
Jun 20 04:25:30 dist-test-slave-nk6d krb5kdc[6463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929530, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Jun 20 04:25:30 dist-test-slave-nk6d krb5kdc[6463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929529, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260620 04:25:30.386322 6547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:30.374220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44244 (local address 127.0.98.62:37461)
0620 04:25:30.374352 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:30.374355 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:30.375290 (+ 935us) server_negotiation.cc:407] Connection header received
0620 04:25:30.376341 (+ 1051us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:30.376345 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:30.376393 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:30.376490 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:30.377545 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:30.378355 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:30.379762 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:30.380088 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:30.380188 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:30.382702 (+ 2514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:30.382706 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:30.382708 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:25:30.382734 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:30.384133 (+ 1399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:30.384815 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:30.384818 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:30.384819 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:30.384888 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:30.385220 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:30.385224 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:30.385226 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:30.385463 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:30.385550 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:30.386042 (+ 492us) server_negotiation.cc:299] Negotiation successful
0620 04:25:30.386157 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 04:25:30.387269 7044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:30.375516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49016 (local address 127.0.98.61:34531)
0620 04:25:30.375847 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:30.375857 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:30.375900 (+ 43us) server_negotiation.cc:407] Connection header received
0620 04:25:30.376145 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:30.376161 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:30.376370 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:30.376540 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:30.377534 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:30.378708 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:30.379862 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:30.380410 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:30.380486 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:30.383036 (+ 2550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:30.383042 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:30.383053 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:25:30.383072 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:30.384831 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:30.385231 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:30.385238 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:30.385240 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:30.385322 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:30.385598 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:30.385602 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:30.385604 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:30.385874 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:30.386060 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:30.386286 (+ 226us) server_negotiation.cc:299] Negotiation successful
0620 04:25:30.386428 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":229,"thread_start_us":91,"threads_started":1}
I20260620 04:25:30.387266 7042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:30.374612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37461 (local address 127.0.0.1:44244)
0620 04:25:30.375119 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:30.375173 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:30.376142 (+ 969us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:30.376553 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:30.376559 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:30.377037 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:30.377406 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:30.377426 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:30.378503 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:30.378506 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:30.379627 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:30.379639 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:30.380237 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:30.380240 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:30.380372 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:30.380922 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:30.380942 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:30.382508 (+ 1566us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:25:30.384272 (+ 1764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:30.384281 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:30.384291 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:30.384616 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:30.384988 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:30.384991 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:30.384993 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:30.385125 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:30.385568 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:25:30.385575 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:25:30.385816 (+ 241us) client_negotiation.cc:769] Sending connection context
0620 04:25:30.385984 (+ 168us) client_negotiation.cc:240] Negotiation successful
0620 04:25:30.386290 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"spinlock_wait_cycles":1920,"thread_start_us":142,"threads_started":1}
I20260620 04:25:30.389173 7040 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34531 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 04:25:30 dist-test-slave-nk6d krb5kdc[6463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929530, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260620 04:25:30.395934 7044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:30.389668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49032 (local address 127.0.98.61:34531)
0620 04:25:30.389781 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:30.389784 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:30.389797 (+ 13us) server_negotiation.cc:407] Connection header received
0620 04:25:30.389844 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:30.389846 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:30.389876 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:30.389946 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:30.390232 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:30.390770 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:30.391425 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:30.391648 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:30.391727 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:30.393496 (+ 1769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:30.393499 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:30.393501 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:25:30.393520 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:30.394644 (+ 1124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:30.394998 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:30.395000 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:30.395002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:30.395068 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:30.395306 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:30.395308 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:30.395309 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:30.395473 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:30.395543 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:30.395729 (+ 186us) server_negotiation.cc:299] Negotiation successful
0620 04:25:30.395815 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 04:25:30.396010 7042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:30.389616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34531 (local address 127.0.0.1:49032)
0620 04:25:30.389711 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:30.389724 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:30.389786 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:30.389956 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:30.389958 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:30.390077 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:30.390159 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:30.390163 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:30.390847 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:30.390849 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:30.391334 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:30.391340 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:30.391723 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:30.391725 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:30.391791 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:30.392223 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:30.392235 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:30.393375 (+ 1140us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:25:30.394738 (+ 1363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:30.394740 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:30.394742 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:30.394908 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:30.395136 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:30.395138 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:30.395139 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:30.395204 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:30.395540 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:25:30.395542 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:25:30.395611 (+ 69us) client_negotiation.cc:769] Sending connection context
0620 04:25:30.395683 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 04:25:30.395774 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260620 04:25:30.399024 7040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1adca5af9a0644fcb8f274cbc0ae7292: Bootstrap starting.
I20260620 04:25:30.401253 7040 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1adca5af9a0644fcb8f274cbc0ae7292: Neither blocks nor log segments found. Creating new log.
I20260620 04:25:30.401890 7040 log.cc:824] T 00000000000000000000000000000000 P 1adca5af9a0644fcb8f274cbc0ae7292: Log is configured to *not* fsync() on all Append() calls
I20260620 04:25:30.403733 7040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1adca5af9a0644fcb8f274cbc0ae7292: No bootstrap required, opened a new log
I20260620 04:25:30.406761 7040 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1adca5af9a0644fcb8f274cbc0ae7292 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "961f73cd6ac541f79ace3f8d78ff255b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37461 } } peers { permanent_uuid: "1adca5af9a0644fcb8f274cbc0ae7292" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34531 } }
I20260620 04:25:30.406961 7040 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1adca5af9a0644fcb8f274cbc0ae7292 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:25:30.407020 7040 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1adca5af9a0644fcb8f274cbc0ae7292 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1adca5af9a0644fcb8f274cbc0ae7292, State: Initialized, Role: FOLLOWER
I20260620 04:25:30.407413 7040 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1adca5af9a0644fcb8f274cbc0ae7292 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "961f73cd6ac541f79ace3f8d78ff255b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37461 } } peers { permanent_uuid: "1adca5af9a0644fcb8f274cbc0ae7292" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34531 } }
I20260620 04:25:30.407832 7046 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1adca5af9a0644fcb8f274cbc0ae7292 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "961f73cd6ac541f79ace3f8d78ff255b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37461 } } peers { permanent_uuid: "1adca5af9a0644fcb8f274cbc0ae7292" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34531 } } }
I20260620 04:25:30.407979 7046 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1adca5af9a0644fcb8f274cbc0ae7292 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 04:25:30.408528 7040 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1adca5af9a0644fcb8f274cbc0ae7292 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 04:25:30.412678 7057 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1adca5af9a0644fcb8f274cbc0ae7292: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 04:25:30.412753 7057 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 04:25:30.414523 6972 master_runner.cc:428] Detected that this master 1adca5af9a0644fcb8f274cbc0ae7292 is joining an existing cluster
I20260620 04:25:30.414577 6972 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:34531
I20260620 04:25:30.423898 7063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:30.416618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37461 (local address 127.0.0.1:44252)
0620 04:25:30.416926 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:30.416939 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:30.417067 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:30.417371 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:30.417373 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:30.417559 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:30.417649 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:30.417654 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:30.418398 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:30.418401 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:30.419009 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:30.419017 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:30.419438 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:30.419442 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:30.419582 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:30.420160 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:30.420173 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:30.420571 (+ 398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:25:30.422287 (+ 1716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:30.422290 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:30.422299 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:30.422506 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:30.422794 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:30.422797 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:30.422799 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:30.422875 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:30.423309 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:25:30.423312 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:25:30.423407 (+ 95us) client_negotiation.cc:769] Sending connection context
0620 04:25:30.423553 (+ 146us) client_negotiation.cc:240] Negotiation successful
0620 04:25:30.423687 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"mutex_wait_us":40,"thread_start_us":95,"threads_started":1}
I20260620 04:25:30.423933 6547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:30.416413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44252 (local address 127.0.98.62:37461)
0620 04:25:30.416526 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:30.416531 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:30.417006 (+ 475us) server_negotiation.cc:407] Connection header received
0620 04:25:30.417165 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:30.417168 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:30.417213 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:30.417277 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:30.417755 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:30.418265 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:30.419114 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:30.419332 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:30.419421 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:30.420688 (+ 1267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:30.420692 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:30.420695 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:25:30.420726 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:30.422200 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:30.422619 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:30.422623 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:30.422625 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:30.422708 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:30.422968 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:30.422973 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:30.422975 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:30.423195 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:30.423294 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:30.423653 (+ 359us) server_negotiation.cc:299] Negotiation successful
0620 04:25:30.423773 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 04:25:30.425058 7044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:30.416410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49034 (local address 127.0.98.61:34531)
0620 04:25:30.416508 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:30.416512 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:30.416988 (+ 476us) server_negotiation.cc:407] Connection header received
0620 04:25:30.417096 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:30.417099 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:30.417138 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:30.417205 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:30.417755 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:30.418519 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:30.419675 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:30.419923 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:30.420047 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:30.421984 (+ 1937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:30.421987 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:30.421990 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:25:30.422012 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:30.423685 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:30.424095 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:30.424098 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:30.424100 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:30.424169 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:30.424397 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:30.424400 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:30.424402 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:30.424550 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:30.424627 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:30.424842 (+ 215us) server_negotiation.cc:299] Negotiation successful
0620 04:25:30.424937 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 04:25:30.425482 7064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:30.416383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34531 (local address 127.0.0.1:49034)
0620 04:25:30.416896 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:30.416909 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:30.417008 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:30.417233 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:30.417236 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:30.417509 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:30.417644 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:30.417652 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:30.418632 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:30.418635 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:30.419568 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:30.419578 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:30.420132 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:30.420135 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:30.420227 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:30.421440 (+ 1213us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:30.421455 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:30.421850 (+ 395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:25:30.423773 (+ 1923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:30.423775 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:30.423778 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:30.423998 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:30.424244 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:30.424247 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:30.424249 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:30.424326 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:30.424628 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:25:30.424631 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:25:30.424706 (+ 75us) client_negotiation.cc:769] Sending connection context
0620 04:25:30.425213 (+ 507us) client_negotiation.cc:240] Negotiation successful
0620 04:25:30.425334 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"mutex_wait_us":23,"thread_start_us":79,"threads_started":1}
Jun 20 04:25:30 dist-test-slave-nk6d krb5kdc[6463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929529, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260620 04:25:30.437795 7044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:30.428991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49040 (local address 127.0.98.61:34531)
0620 04:25:30.429102 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:30.429106 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:30.429882 (+ 776us) server_negotiation.cc:407] Connection header received
0620 04:25:30.430049 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:30.430052 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:30.430115 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:30.430185 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:30.430906 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:30.431449 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:30.432655 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:30.432855 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:30.432936 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:30.435033 (+ 2097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:30.435036 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:30.435038 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:25:30.435056 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:30.436274 (+ 1218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:30.436681 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:30.436685 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:30.436686 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:30.436749 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:30.436979 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:30.436982 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:30.436984 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:30.437131 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:30.437210 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:30.437541 (+ 331us) server_negotiation.cc:299] Negotiation successful
0620 04:25:30.437640 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 04:25:30.437799 7065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:30.428974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34531 (local address 127.0.0.1:49040)
0620 04:25:30.429755 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:30.429812 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:30.429937 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:30.430227 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:30.430233 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:30.430656 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:30.430774 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:30.430781 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:30.431536 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:30.431539 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:30.432551 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:30.432562 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:30.432945 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:30.432949 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:30.433052 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:30.433494 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:30.433508 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:30.434913 (+ 1405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:25:30.436365 (+ 1452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:30.436374 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:30.436379 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:30.436596 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:30.436825 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:30.436828 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:30.436830 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:30.436904 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:30.437224 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:25:30.437231 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:25:30.437339 (+ 108us) client_negotiation.cc:769] Sending connection context
0620 04:25:30.437467 (+ 128us) client_negotiation.cc:240] Negotiation successful
0620 04:25:30.437646 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"thread_start_us":85,"threads_started":1}
I20260620 04:25:30.439083 6508 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:34531: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "1adca5af9a0644fcb8f274cbc0ae7292"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 34531
}
attrs {
promote: true
}
}
dest_uuid: "961f73cd6ac541f79ace3f8d78ff255b"
cas_config_opid_index: -1
I20260620 04:25:30.439702 6508 consensus_queue.cc:237] T 00000000000000000000000000000000 P 961f73cd6ac541f79ace3f8d78ff255b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "961f73cd6ac541f79ace3f8d78ff255b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37461 } } peers { permanent_uuid: "1adca5af9a0644fcb8f274cbc0ae7292" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34531 } attrs { promote: true } }
I20260620 04:25:30.441615 7067 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 961f73cd6ac541f79ace3f8d78ff255b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 1adca5af9a0644fcb8f274cbc0ae7292 (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "961f73cd6ac541f79ace3f8d78ff255b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37461 } } peers { permanent_uuid: "1adca5af9a0644fcb8f274cbc0ae7292" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34531 } attrs { promote: true } } }
I20260620 04:25:30.442200 7013 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1adca5af9a0644fcb8f274cbc0ae7292 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:25:30.443231 7067 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:34531
I20260620 04:25:30.443481 7013 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1adca5af9a0644fcb8f274cbc0ae7292 [term 1 FOLLOWER]: Refusing update from remote peer 961f73cd6ac541f79ace3f8d78ff255b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260620 04:25:30.443605 7066 sys_catalog.cc:455] T 00000000000000000000000000000000 P 961f73cd6ac541f79ace3f8d78ff255b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "961f73cd6ac541f79ace3f8d78ff255b" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "961f73cd6ac541f79ace3f8d78ff255b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37461 } } peers { permanent_uuid: "1adca5af9a0644fcb8f274cbc0ae7292" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34531 } attrs { promote: true } } }
I20260620 04:25:30.443701 7066 sys_catalog.cc:458] T 00000000000000000000000000000000 P 961f73cd6ac541f79ace3f8d78ff255b [sys.catalog]: This master's current role is: LEADER
I20260620 04:25:30.444172 7066 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 961f73cd6ac541f79ace3f8d78ff255b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1adca5af9a0644fcb8f274cbc0ae7292" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34531 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:25:30.444356 6972 master.cc:537] Master@127.0.98.61:34531 shutting down...
W20260620 04:25:30.445866 6493 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260620 04:25:30.446746 6493 consensus_peers.cc:487] T 00000000000000000000000000000000 P 961f73cd6ac541f79ace3f8d78ff255b -> Peer 1adca5af9a0644fcb8f274cbc0ae7292 (127.0.98.61:34531): Couldn't send request to peer 1adca5af9a0644fcb8f274cbc0ae7292. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
I20260620 04:25:30.454030 6972 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 1adca5af9a0644fcb8f274cbc0ae7292 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:25:30.454321 6972 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 1adca5af9a0644fcb8f274cbc0ae7292 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:25:30.454416 6972 tablet_replica.cc:330] T 00000000000000000000000000000000 P 1adca5af9a0644fcb8f274cbc0ae7292: stopping tablet replica
I20260620 04:25:30.458915 6972 master.cc:552] Master@127.0.98.61:34531 shutdown complete.
I20260620 04:25:30.458963 6972 master_runner.cc:305] Clearing existing system tablet
I20260620 04:25:30.459163 6972 fs_manager.cc:359] Metadata directory not provided
I20260620 04:25:30.459208 6972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:25:30.460178 6972 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 04:25:30.460866 7073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:30.461058 6972 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:25:30.461176 6972 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 04:25:30.461211 6972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/master-1/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/master-1/wal
uuid: "1adca5af9a0644fcb8f274cbc0ae7292"
format_stamp: "Formatted at 2026-06-20 04:25:30 on dist-test-slave-nk6d"
server_key: "f40d2cf75d6a7763f4f86bd99f81129a"
server_key_iv: "5e11fd9ece9550cf1eacf0985bc5e9ea"
server_key_version: "encryptionkey@0"
I20260620 04:25:30.462777 6972 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 1adca5af9a0644fcb8f274cbc0ae7292: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:25:30.468357 6972 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 1adca5af9a0644fcb8f274cbc0ae7292: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 04:25:30.468442 6972 log.cc:1196] T 00000000000000000000000000000000 P 1adca5af9a0644fcb8f274cbc0ae7292: Deleting WAL directory at /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260620 04:25:30.468773 6972 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 1adca5af9a0644fcb8f274cbc0ae7292: Deleting consensus metadata
I20260620 04:25:30.468994 6972 master_runner.cc:315] Copying system tablet from 127.0.98.62:37461
I20260620 04:25:30.470057 6972 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 1adca5af9a0644fcb8f274cbc0ae7292: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:37461
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260620 04:25:30.475481 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:45713
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37461,127.0.98.61:34531,127.0.98.60:45713
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 04:25:30.478456 7079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:30.470538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37461 (local address 127.0.0.1:44254)
0620 04:25:30.470903 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:30.470919 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:30.471052 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:30.471301 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:30.471304 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:30.471514 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:30.471651 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:30.471659 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:30.472609 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:30.472613 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:30.473538 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:30.473550 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:30.473983 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:30.473988 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:30.474086 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:30.474563 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:30.474580 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:30.475007 (+ 427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:25:30.477081 (+ 2074us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:30.477085 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:30.477087 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:30.477272 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:30.477515 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:30.477518 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:30.477520 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:30.477596 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:30.477924 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:25:30.477927 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:25:30.478023 (+ 96us) client_negotiation.cc:769] Sending connection context
0620 04:25:30.478120 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 04:25:30.478249 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":82,"threads_started":1}
I20260620 04:25:30.478498 6547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:30.470664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44254 (local address 127.0.98.62:37461)
0620 04:25:30.470804 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:30.470809 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:30.471020 (+ 211us) server_negotiation.cc:407] Connection header received
0620 04:25:30.471148 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:30.471150 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:30.471195 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:30.471261 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:30.471810 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:30.472503 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:30.473666 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:30.473874 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:30.473952 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:30.475151 (+ 1199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:30.475155 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:30.475157 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:25:30.475184 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:30.476986 (+ 1802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:30.477357 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:30.477361 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:30.477363 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:30.477439 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:30.477670 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:30.477673 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:30.477675 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:30.477842 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:30.477931 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:30.478202 (+ 271us) server_negotiation.cc:299] Negotiation successful
0620 04:25:30.478327 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 04:25:30.478957 6528 tablet_copy_service.cc:140] P 961f73cd6ac541f79ace3f8d78ff255b: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 1adca5af9a0644fcb8f274cbc0ae7292 ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:44254)
I20260620 04:25:30.479128 6528 tablet_copy_service.cc:161] P 961f73cd6ac541f79ace3f8d78ff255b: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 1adca5af9a0644fcb8f274cbc0ae7292 at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:44254: session id = 1adca5af9a0644fcb8f274cbc0ae7292-00000000000000000000000000000000
I20260620 04:25:30.480995 6528 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 961f73cd6ac541f79ace3f8d78ff255b: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 04:25:30.482667 6972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 04:25:30.484488 6972 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 1adca5af9a0644fcb8f274cbc0ae7292: tablet copy: Starting download of 0 data blocks...
I20260620 04:25:30.484637 6972 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 1adca5af9a0644fcb8f274cbc0ae7292: tablet copy: Starting download of 1 WAL segments...
I20260620 04:25:30.486079 6972 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 1adca5af9a0644fcb8f274cbc0ae7292: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 04:25:30.487550 6528 tablet_copy_service.cc:342] P 961f73cd6ac541f79ace3f8d78ff255b: Request end of tablet copy session 1adca5af9a0644fcb8f274cbc0ae7292-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:44254
I20260620 04:25:30.487645 6528 tablet_copy_service.cc:433] P 961f73cd6ac541f79ace3f8d78ff255b: ending tablet copy session 1adca5af9a0644fcb8f274cbc0ae7292-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 1adca5af9a0644fcb8f274cbc0ae7292
I20260620 04:25:30.494454 6972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:25:30.496052 7081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:25:30.496115 6972 server_base.cc:1034] running on GCE node
W20260620 04:25:30.496236 7082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:25:30.496131 7084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:25:30.496595 6972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:25:30.496850 6972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:25:30.497982 6972 hybrid_clock.cc:648] HybridClock initialized: now 1781929530497975 us; error 24 us; skew 500 ppm
Jun 20 04:25:30 dist-test-slave-nk6d krb5kdc[6463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929530, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:25:30.499269 6972 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260620 04:25:30.499683 6972 webserver.cc:460] Webserver started at http://127.0.98.61:34733/ using document root <none> and password file <none>
I20260620 04:25:30.499848 6972 fs_manager.cc:359] Metadata directory not provided
I20260620 04:25:30.499882 6972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:25:30.500902 6972 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 04:25:30.501688 7091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:30.501890 6972 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 04:25:30.501957 6972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/master-1/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/master-1/wal
uuid: "1adca5af9a0644fcb8f274cbc0ae7292"
format_stamp: "Formatted at 2026-06-20 04:25:30 on dist-test-slave-nk6d"
server_key: "f40d2cf75d6a7763f4f86bd99f81129a"
server_key_iv: "5e11fd9ece9550cf1eacf0985bc5e9ea"
server_key_version: "encryptionkey@0"
I20260620 04:25:30.502036 6972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:25:30.517756 6972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:25:30.519487 6972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:25:30.523885 6972 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:34531
I20260620 04:25:30.523901 7143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:34531 every 8 connection(s)
I20260620 04:25:30.524435 6972 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/master-1/data/info.pb
I20260620 04:25:30.525759 7144 sys_catalog.cc:263] Verifying existing consensus state
I20260620 04:25:30.526580 7144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1adca5af9a0644fcb8f274cbc0ae7292: Bootstrap starting.
I20260620 04:25:30.534353 7144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1adca5af9a0644fcb8f274cbc0ae7292: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 04:25:30.534590 7144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1adca5af9a0644fcb8f274cbc0ae7292: Bootstrap complete.
I20260620 04:25:30.535030 7144 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1adca5af9a0644fcb8f274cbc0ae7292 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "961f73cd6ac541f79ace3f8d78ff255b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37461 } } peers { permanent_uuid: "1adca5af9a0644fcb8f274cbc0ae7292" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34531 } attrs { promote: true } }
I20260620 04:25:30.535136 7144 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1adca5af9a0644fcb8f274cbc0ae7292 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1adca5af9a0644fcb8f274cbc0ae7292, State: Initialized, Role: LEARNER
I20260620 04:25:30.535228 7144 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1adca5af9a0644fcb8f274cbc0ae7292 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "961f73cd6ac541f79ace3f8d78ff255b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37461 } } peers { permanent_uuid: "1adca5af9a0644fcb8f274cbc0ae7292" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34531 } attrs { promote: true } }
I20260620 04:25:30.535557 7147 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1adca5af9a0644fcb8f274cbc0ae7292 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "961f73cd6ac541f79ace3f8d78ff255b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37461 } } peers { permanent_uuid: "1adca5af9a0644fcb8f274cbc0ae7292" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34531 } attrs { promote: true } } }
I20260620 04:25:30.535638 7144 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1adca5af9a0644fcb8f274cbc0ae7292 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:25:30.535723 7147 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1adca5af9a0644fcb8f274cbc0ae7292 [sys.catalog]: This master's current role is: LEARNER
I20260620 04:25:30.538636 7158 catalog_manager.cc:1259] Loaded cluster ID: 65e72dfb217a4b24b6c29af381fead6f
I20260620 04:25:30.538738 7158 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1adca5af9a0644fcb8f274cbc0ae7292: loading cluster ID for follower catalog manager: success
I20260620 04:25:30.540324 7158 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1adca5af9a0644fcb8f274cbc0ae7292: acquiring CA information for follower catalog manager: success
I20260620 04:25:30.541501 7158 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1adca5af9a0644fcb8f274cbc0ae7292: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 04:25:30.624926 7080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:25:30.625217 7080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:25:30.625257 7080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:25:30.629283 7080 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:25:30.629345 7080 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:25:30.629364 7080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:25:30.629377 7080 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:25:30.629391 7080 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:25:30.634647 7080 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37461,127.0.98.61:34531,127.0.98.60:45713
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:45713
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7080
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:25:30.635699 7080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:25:30.636926 7080 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:25:30.645128 7166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:25:30.645177 7163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:25:30.645218 7080 server_base.cc:1034] running on GCE node
W20260620 04:25:30.645136 7164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:25:30.646001 7080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:25:30.647131 7080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:25:30.648305 7080 hybrid_clock.cc:648] HybridClock initialized: now 1781929530648291 us; error 25 us; skew 500 ppm
Jun 20 04:25:30 dist-test-slave-nk6d krb5kdc[6463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929530, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:25:30.650840 7080 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260620 04:25:30.651724 7080 webserver.cc:460] Webserver started at http://127.0.98.60:42511/ using document root <none> and password file <none>
I20260620 04:25:30.652294 7080 fs_manager.cc:359] Metadata directory not provided
I20260620 04:25:30.652360 7080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:25:30.652549 7080 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:25:30.654510 7080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/master-2/data/instance:
uuid: "3b026c99d08842f5a61f0a658e08398a"
format_stamp: "Formatted at 2026-06-20 04:25:30 on dist-test-slave-nk6d"
server_key: "344a51a282ec179295ac72a13be41482"
server_key_iv: "190bae46636527da7d75986f7e2a2221"
server_key_version: "encryptionkey@0"
I20260620 04:25:30.655021 7080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/master-2/wal/instance:
uuid: "3b026c99d08842f5a61f0a658e08398a"
format_stamp: "Formatted at 2026-06-20 04:25:30 on dist-test-slave-nk6d"
server_key: "344a51a282ec179295ac72a13be41482"
server_key_iv: "190bae46636527da7d75986f7e2a2221"
server_key_version: "encryptionkey@0"
I20260620 04:25:30.658613 7080 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260620 04:25:30.661161 7173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:30.662453 7080 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 04:25:30.662578 7080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/master-2/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/master-2/wal
uuid: "3b026c99d08842f5a61f0a658e08398a"
format_stamp: "Formatted at 2026-06-20 04:25:30 on dist-test-slave-nk6d"
server_key: "344a51a282ec179295ac72a13be41482"
server_key_iv: "190bae46636527da7d75986f7e2a2221"
server_key_version: "encryptionkey@0"
I20260620 04:25:30.662675 7080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:25:30.687130 7080 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:25:30.690173 7080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:25:30.690373 7080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:25:30.697788 7080 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:45713
I20260620 04:25:30.698012 7225 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:45713 every 8 connection(s)
I20260620 04:25:30.698776 7080 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/master-2/data/info.pb
I20260620 04:25:30.701442 7226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:25:30.702881 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 7080
I20260620 04:25:30.706717 7226 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37461 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 04:25:30 dist-test-slave-nk6d krb5kdc[6463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929529, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Jun 20 04:25:30 dist-test-slave-nk6d krb5kdc[6463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929530, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 04:25:30.716756 6547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:30.708433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44258 (local address 127.0.98.62:37461)
0620 04:25:30.708549 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:30.708554 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:30.708916 (+ 362us) server_negotiation.cc:407] Connection header received
0620 04:25:30.709039 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:30.709042 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:30.709084 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:30.709146 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:30.709878 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:30.710369 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:30.711360 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:30.711579 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:30.711664 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:30.713672 (+ 2008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:30.713675 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:30.713677 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:25:30.713698 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:30.715025 (+ 1327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:30.715363 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:30.715366 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:30.715367 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:30.715433 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:30.715695 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:30.715698 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:30.715700 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:30.715897 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:30.716119 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:30.716505 (+ 386us) server_negotiation.cc:299] Negotiation successful
0620 04:25:30.716600 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 04:25:30.717195 7228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:30.704554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44084 (local address 127.0.98.60:45713)
0620 04:25:30.705031 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:30.705049 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:30.705084 (+ 35us) server_negotiation.cc:407] Connection header received
0620 04:25:30.705857 (+ 773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:30.705884 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:30.706257 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:30.706562 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:30.707359 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:30.708572 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:30.709672 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:30.710148 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:30.710284 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:30.712733 (+ 2449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:30.712738 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:30.712747 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:25:30.712767 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:30.714291 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:30.714741 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:30.714745 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:30.714750 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:30.714852 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:30.715373 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:30.715376 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:30.715377 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:30.715731 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:30.715884 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:30.716163 (+ 279us) server_negotiation.cc:299] Negotiation successful
0620 04:25:30.716428 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":157,"threads_started":1}
I20260620 04:25:30.717192 7229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:30.708319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37461 (local address 127.0.0.1:44258)
0620 04:25:30.708778 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:30.708827 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:30.708942 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:30.709205 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:30.709210 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:30.709620 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:30.709763 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:30.709769 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:30.710481 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:30.710484 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:30.711247 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:30.711256 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:30.711705 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:30.711709 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:30.711811 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:30.712272 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:30.712294 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:30.713549 (+ 1255us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:25:30.715102 (+ 1553us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:30.715106 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:30.715111 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:30.715289 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:30.715532 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:30.715534 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:30.715536 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:30.715607 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:30.716165 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:25:30.716172 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:25:30.716300 (+ 128us) client_negotiation.cc:769] Sending connection context
0620 04:25:30.716433 (+ 133us) client_negotiation.cc:240] Negotiation successful
0620 04:25:30.716598 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":128,"threads_started":1}
I20260620 04:25:30.718323 7226 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34531 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 04:25:30 dist-test-slave-nk6d krb5kdc[6463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929530, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260620 04:25:30.726377 7229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:30.718735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34531 (local address 127.0.0.1:49044)
0620 04:25:30.718833 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:30.718846 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:30.718908 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:30.719616 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:30.719619 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:30.719760 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:30.720177 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:30.720184 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:30.721013 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:30.721016 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:30.721531 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:30.721538 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:30.721930 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:30.721933 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:30.722018 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:30.722413 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:30.722423 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:30.723590 (+ 1167us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:25:30.725157 (+ 1567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:30.725160 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:30.725162 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:30.725329 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:30.725558 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:30.725560 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:30.725562 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:30.725635 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:30.725950 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:25:30.725952 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:25:30.726034 (+ 82us) client_negotiation.cc:769] Sending connection context
0620 04:25:30.726114 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 04:25:30.726227 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260620 04:25:30.726415 7230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:30.719164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49044 (local address 127.0.98.61:34531)
0620 04:25:30.719415 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:30.719419 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:30.719430 (+ 11us) server_negotiation.cc:407] Connection header received
0620 04:25:30.719464 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:30.719466 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:30.719507 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:30.719589 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:30.720283 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:30.720917 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:30.721630 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:30.721843 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:30.721923 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:30.723711 (+ 1788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:30.723714 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:30.723716 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:25:30.723738 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:30.725070 (+ 1332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:30.725408 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:30.725411 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:30.725413 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:30.725482 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:30.725706 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:30.725709 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:30.725711 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:30.725871 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:30.725949 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:30.726168 (+ 219us) server_negotiation.cc:299] Negotiation successful
0620 04:25:30.726273 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":81,"threads_started":1}
I20260620 04:25:30.727397 7226 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45713 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 04:25:30 dist-test-slave-nk6d krb5kdc[6463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929530, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260620 04:25:30.734208 7229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:30.727798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45713 (local address 127.0.0.1:44086)
0620 04:25:30.727979 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:30.727991 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:30.728062 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:30.728235 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:30.728236 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:30.728346 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:30.728426 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:30.728431 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:30.729177 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:30.729179 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:30.729780 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:30.729788 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:30.730174 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:30.730177 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:30.730227 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:30.730636 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:30.730647 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:30.731672 (+ 1025us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:25:30.733013 (+ 1341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:30.733015 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:30.733016 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:30.733173 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:30.733353 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:30.733355 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:30.733356 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:30.733414 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:30.733745 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:25:30.733747 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:25:30.733837 (+ 90us) client_negotiation.cc:769] Sending connection context
0620 04:25:30.733934 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 04:25:30.734045 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97}
I20260620 04:25:30.734208 7228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:30.727858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44086 (local address 127.0.98.60:45713)
0620 04:25:30.727984 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:30.727988 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:30.728029 (+ 41us) server_negotiation.cc:407] Connection header received
0620 04:25:30.728115 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:30.728118 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:30.728154 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:30.728228 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:30.728525 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:30.729102 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:30.729867 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:30.730097 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:30.730171 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:30.731773 (+ 1602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:30.731776 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:30.731777 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:25:30.731796 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:30.732924 (+ 1128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:30.733239 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:30.733241 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:30.733242 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:30.733299 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:30.733461 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:30.733462 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:30.733463 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:30.733597 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:30.733660 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:30.733992 (+ 332us) server_negotiation.cc:299] Negotiation successful
0620 04:25:30.734094 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":14,"server-negotiator.queue_time_us":41}
I20260620 04:25:30.737141 7226 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b026c99d08842f5a61f0a658e08398a: Bootstrap starting.
I20260620 04:25:30.739307 7226 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b026c99d08842f5a61f0a658e08398a: Neither blocks nor log segments found. Creating new log.
I20260620 04:25:30.740053 7226 log.cc:824] T 00000000000000000000000000000000 P 3b026c99d08842f5a61f0a658e08398a: Log is configured to *not* fsync() on all Append() calls
I20260620 04:25:30.742368 7226 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b026c99d08842f5a61f0a658e08398a: No bootstrap required, opened a new log
I20260620 04:25:30.745289 7226 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b026c99d08842f5a61f0a658e08398a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "961f73cd6ac541f79ace3f8d78ff255b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37461 } } peers { permanent_uuid: "1adca5af9a0644fcb8f274cbc0ae7292" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34531 } } peers { permanent_uuid: "3b026c99d08842f5a61f0a658e08398a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45713 } }
I20260620 04:25:30.745491 7226 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3b026c99d08842f5a61f0a658e08398a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:25:30.745548 7226 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b026c99d08842f5a61f0a658e08398a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b026c99d08842f5a61f0a658e08398a, State: Initialized, Role: FOLLOWER
I20260620 04:25:30.745949 7226 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b026c99d08842f5a61f0a658e08398a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "961f73cd6ac541f79ace3f8d78ff255b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37461 } } peers { permanent_uuid: "1adca5af9a0644fcb8f274cbc0ae7292" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34531 } } peers { permanent_uuid: "3b026c99d08842f5a61f0a658e08398a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45713 } }
I20260620 04:25:30.746374 7232 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b026c99d08842f5a61f0a658e08398a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "961f73cd6ac541f79ace3f8d78ff255b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37461 } } peers { permanent_uuid: "1adca5af9a0644fcb8f274cbc0ae7292" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34531 } } peers { permanent_uuid: "3b026c99d08842f5a61f0a658e08398a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45713 } } }
I20260620 04:25:30.746485 7232 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b026c99d08842f5a61f0a658e08398a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 04:25:30.747043 7226 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b026c99d08842f5a61f0a658e08398a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:25:30.748099 6699 heartbeater.cc:503] Master 127.0.98.62:37461 was elected leader, sending a full tablet report...
W20260620 04:25:30.751518 7243 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3b026c99d08842f5a61f0a658e08398a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 04:25:30.751590 7243 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 04:25:30.754081 7080 master_runner.cc:428] Detected that this master 3b026c99d08842f5a61f0a658e08398a is joining an existing cluster
I20260620 04:25:30.754127 7080 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:45713
I20260620 04:25:30.764905 7230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:30.756012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49056 (local address 127.0.98.61:34531)
0620 04:25:30.756144 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:30.756149 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:30.756419 (+ 270us) server_negotiation.cc:407] Connection header received
0620 04:25:30.756553 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:30.756556 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:30.756605 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:30.756656 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:30.757170 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:30.757893 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:30.759768 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:30.760116 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:30.760183 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:30.761506 (+ 1323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:30.761510 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:30.761513 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:25:30.761544 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:30.763160 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:30.763691 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:30.763695 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:30.763697 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:30.763785 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:30.764134 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:30.764138 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:30.764140 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:30.764343 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:30.764425 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:30.764655 (+ 230us) server_negotiation.cc:299] Negotiation successful
0620 04:25:30.764771 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 04:25:30.764911 7249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:30.756121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34531 (local address 127.0.0.1:49056)
0620 04:25:30.756344 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:30.756358 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:30.756450 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:30.756788 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:30.756790 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:30.756963 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:30.757060 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:30.757065 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:30.757997 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:30.758000 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:30.759653 (+ 1653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:30.759660 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:30.760345 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:30.760349 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:30.760435 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:30.761012 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:30.761023 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:30.761369 (+ 346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:25:30.763281 (+ 1912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:30.763285 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:30.763287 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:30.763582 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:30.763879 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:30.763882 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:30.763884 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:30.764023 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:30.764423 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:25:30.764426 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:25:30.764514 (+ 88us) client_negotiation.cc:769] Sending connection context
0620 04:25:30.764596 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 04:25:30.764722 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"mutex_wait_us":37,"spinlock_wait_cycles":5504,"thread_start_us":76,"threads_started":1}
I20260620 04:25:30.765503 7228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:30.756590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44102 (local address 127.0.98.60:45713)
0620 04:25:30.756680 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:30.756684 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:30.757488 (+ 804us) server_negotiation.cc:407] Connection header received
0620 04:25:30.757627 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:30.757630 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:30.757667 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:30.757712 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:30.758304 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:30.758980 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:30.760510 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:30.760816 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:30.760898 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:30.762398 (+ 1500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:30.762401 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:30.762404 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:25:30.762424 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:30.763877 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:30.764273 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:30.764277 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:30.764279 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:30.764364 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:30.764654 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:30.764657 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:30.764659 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:30.764860 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:30.764930 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:30.765230 (+ 300us) server_negotiation.cc:299] Negotiation successful
0620 04:25:30.765339 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":37}
I20260620 04:25:30.765947 7250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:30.756542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45713 (local address 127.0.0.1:44102)
0620 04:25:30.757421 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:30.757434 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:30.757536 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:30.757846 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:30.757849 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:30.758071 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:30.758182 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:30.758190 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:30.759078 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:30.759081 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:30.760381 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:30.760392 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:30.761042 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:30.761046 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:30.761141 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:30.761871 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:30.761887 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:30.762277 (+ 390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:25:30.763955 (+ 1678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:30.763958 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:30.763984 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:30.764186 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:30.764470 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:30.764473 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:30.764475 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:30.764564 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:30.764992 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:25:30.764994 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:25:30.765073 (+ 79us) client_negotiation.cc:769] Sending connection context
0620 04:25:30.765635 (+ 562us) client_negotiation.cc:240] Negotiation successful
0620 04:25:30.765755 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":812,"mutex_wait_us":1,"spinlock_wait_cycles":6016,"thread_start_us":76,"threads_started":1}
I20260620 04:25:30.770360 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 6570
I20260620 04:25:30.777376 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:42601
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:37461
--webserver_port=33081
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:37461,127.0.98.61:34531
--tserver_master_addrs=127.0.98.62:37461,127.0.98.61:34531,127.0.98.60:45713 with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 04:25:30.788962 7251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:30.756186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37461 (local address 127.0.0.1:44262)
0620 04:25:30.756930 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:30.756949 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:30.757222 (+ 273us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:30.757560 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:30.757563 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:30.757766 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:30.757862 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:30.757868 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:30.758896 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:30.758899 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:30.759772 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:30.759779 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:30.760379 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:30.760382 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:30.760474 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:30.784534 (+ 24060us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:30.784553 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:30.785031 (+ 478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:25:30.787257 (+ 2226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:30.787261 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:30.787263 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:30.787519 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:30.787857 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:30.787860 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:30.787862 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:30.787952 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:30.788410 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:25:30.788412 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:25:30.788519 (+ 107us) client_negotiation.cc:769] Sending connection context
0620 04:25:30.788621 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 04:25:30.788770 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":650,"mutex_wait_us":1,"thread_start_us":96,"threads_started":1}
I20260620 04:25:30.789032 6547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:30.755988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44262 (local address 127.0.98.62:37461)
0620 04:25:30.756090 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:30.756095 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:30.757154 (+ 1059us) server_negotiation.cc:407] Connection header received
0620 04:25:30.757325 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:30.757328 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:30.757375 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:30.757457 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:30.757996 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:30.758772 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:30.759872 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:30.760200 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:30.760260 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:30.785259 (+ 24999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:30.785264 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:30.785268 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:25:30.785304 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:30.787124 (+ 1820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:30.787653 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:30.787657 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:30.787659 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:30.787757 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:30.788053 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:30.788056 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:30.788058 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:30.788308 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:30.788412 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:30.788719 (+ 307us) server_negotiation.cc:299] Negotiation successful
0620 04:25:30.788857 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
Jun 20 04:25:30 dist-test-slave-nk6d krb5kdc[6463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929529, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260620 04:25:30.802243 7065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:30.793153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45713 (local address 127.0.0.1:44116)
0620 04:25:30.793348 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:30.793362 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:30.793435 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:30.793818 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:30.793821 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:30.794005 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:30.794115 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:30.794123 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:30.795106 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:30.795110 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:30.795989 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:30.796001 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:30.796590 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:30.796595 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:30.796700 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:30.797177 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:30.797193 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:30.798673 (+ 1480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:25:30.800668 (+ 1995us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:30.800672 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:30.800674 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:30.800915 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:30.801190 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:30.801193 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:30.801195 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:30.801282 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:30.801676 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:25:30.801679 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:25:30.801774 (+ 95us) client_negotiation.cc:769] Sending connection context
0620 04:25:30.801874 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 04:25:30.802013 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111}
I20260620 04:25:30.802736 7228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:30.793218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44116 (local address 127.0.98.60:45713)
0620 04:25:30.793564 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:30.793568 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:30.793578 (+ 10us) server_negotiation.cc:407] Connection header received
0620 04:25:30.793639 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:30.793641 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:30.793687 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:30.793758 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:30.794231 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:30.795000 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:30.796136 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:30.796478 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:30.796570 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:30.798838 (+ 2268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:30.798842 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:30.798845 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:25:30.798873 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:30.800566 (+ 1693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:30.801009 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:30.801012 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:30.801014 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:30.801099 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:30.801368 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:30.801371 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:30.801372 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:30.801583 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:30.801683 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:30.802480 (+ 797us) server_negotiation.cc:299] Negotiation successful
0620 04:25:30.802591 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282}
I20260620 04:25:30.803690 6508 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:45713: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "3b026c99d08842f5a61f0a658e08398a"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 45713
}
attrs {
promote: true
}
}
dest_uuid: "961f73cd6ac541f79ace3f8d78ff255b"
cas_config_opid_index: 5
I20260620 04:25:30.804327 6508 consensus_queue.cc:237] T 00000000000000000000000000000000 P 961f73cd6ac541f79ace3f8d78ff255b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "961f73cd6ac541f79ace3f8d78ff255b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37461 } } peers { permanent_uuid: "1adca5af9a0644fcb8f274cbc0ae7292" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34531 } attrs { promote: true } } peers { permanent_uuid: "3b026c99d08842f5a61f0a658e08398a" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 45713 } attrs { promote: true } }
I20260620 04:25:30.805464 7066 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 961f73cd6ac541f79ace3f8d78ff255b [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 3b026c99d08842f5a61f0a658e08398a (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "961f73cd6ac541f79ace3f8d78ff255b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37461 } } peers { permanent_uuid: "1adca5af9a0644fcb8f274cbc0ae7292" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34531 } attrs { promote: true } } peers { permanent_uuid: "3b026c99d08842f5a61f0a658e08398a" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 45713 } attrs { promote: true } } }
I20260620 04:25:30.806648 7200 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b026c99d08842f5a61f0a658e08398a [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:25:30.806906 7066 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:45713
I20260620 04:25:30.807085 7066 sys_catalog.cc:455] T 00000000000000000000000000000000 P 961f73cd6ac541f79ace3f8d78ff255b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "961f73cd6ac541f79ace3f8d78ff255b" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "961f73cd6ac541f79ace3f8d78ff255b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37461 } } peers { permanent_uuid: "1adca5af9a0644fcb8f274cbc0ae7292" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34531 } attrs { promote: true } } peers { permanent_uuid: "3b026c99d08842f5a61f0a658e08398a" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 45713 } attrs { promote: true } } }
I20260620 04:25:30.807173 7066 sys_catalog.cc:458] T 00000000000000000000000000000000 P 961f73cd6ac541f79ace3f8d78ff255b [sys.catalog]: This master's current role is: LEADER
I20260620 04:25:30.807713 7080 master.cc:537] Master@127.0.98.60:45713 shutting down...
I20260620 04:25:30.808032 7200 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3b026c99d08842f5a61f0a658e08398a [term 1 FOLLOWER]: Refusing update from remote peer 961f73cd6ac541f79ace3f8d78ff255b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 04:25:30.809269 7067 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 961f73cd6ac541f79ace3f8d78ff255b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b026c99d08842f5a61f0a658e08398a" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 45713 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 04:25:30.811527 6495 consensus_peers.cc:487] T 00000000000000000000000000000000 P 961f73cd6ac541f79ace3f8d78ff255b -> Peer 3b026c99d08842f5a61f0a658e08398a (127.0.98.60:45713): Couldn't send request to peer 3b026c99d08842f5a61f0a658e08398a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 04:25:30.813503 7230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:30.805914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49058 (local address 127.0.98.61:34531)
0620 04:25:30.806022 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:30.806026 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:30.808817 (+ 2791us) server_negotiation.cc:407] Connection header received
0620 04:25:30.809012 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:30.809015 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:30.809073 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:30.809133 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:25:30.809583 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:30.810369 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:30.811992 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:30.812632 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:30.812709 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:30.812845 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:30.813298 (+ 453us) server_negotiation.cc:299] Negotiation successful
0620 04:25:30.813379 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 04:25:30.814019 7065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:30.805613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34531 (local address 127.0.0.1:49058)
0620 04:25:30.808749 (+ 3136us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:30.808762 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:30.808897 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:30.809274 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:30.809277 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:30.809293 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:25:30.809372 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:30.809377 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:30.810499 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:30.810502 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:30.811824 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:30.811835 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:30.813044 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:30.813048 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:30.813138 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:30.813180 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 04:25:30.813814 (+ 634us) client_negotiation.cc:240] Negotiation successful
0620 04:25:30.813898 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3067}
I20260620 04:25:30.814596 7115 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1adca5af9a0644fcb8f274cbc0ae7292 [term 1 LEARNER]: Refusing update from remote peer 961f73cd6ac541f79ace3f8d78ff255b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 04:25:30.815007 7067 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 961f73cd6ac541f79ace3f8d78ff255b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1adca5af9a0644fcb8f274cbc0ae7292" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34531 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 04:25:30.816748 7115 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1adca5af9a0644fcb8f274cbc0ae7292 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 3b026c99d08842f5a61f0a658e08398a (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "961f73cd6ac541f79ace3f8d78ff255b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37461 } } peers { permanent_uuid: "1adca5af9a0644fcb8f274cbc0ae7292" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34531 } attrs { promote: true } } peers { permanent_uuid: "3b026c99d08842f5a61f0a658e08398a" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 45713 } attrs { promote: true } } }
I20260620 04:25:30.818819 7147 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1adca5af9a0644fcb8f274cbc0ae7292 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 961f73cd6ac541f79ace3f8d78ff255b. Latest consensus state: current_term: 1 leader_uuid: "961f73cd6ac541f79ace3f8d78ff255b" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "961f73cd6ac541f79ace3f8d78ff255b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37461 } } peers { permanent_uuid: "1adca5af9a0644fcb8f274cbc0ae7292" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34531 } attrs { promote: true } } peers { permanent_uuid: "3b026c99d08842f5a61f0a658e08398a" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 45713 } attrs { promote: true } } }
I20260620 04:25:30.818931 7147 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1adca5af9a0644fcb8f274cbc0ae7292 [sys.catalog]: This master's current role is: LEARNER
I20260620 04:25:30.819010 7147 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1adca5af9a0644fcb8f274cbc0ae7292 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "961f73cd6ac541f79ace3f8d78ff255b" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "961f73cd6ac541f79ace3f8d78ff255b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37461 } } peers { permanent_uuid: "1adca5af9a0644fcb8f274cbc0ae7292" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34531 } attrs { promote: true } } peers { permanent_uuid: "3b026c99d08842f5a61f0a658e08398a" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 45713 } attrs { promote: true } } }
I20260620 04:25:30.819068 7147 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1adca5af9a0644fcb8f274cbc0ae7292 [sys.catalog]: This master's current role is: LEARNER
I20260620 04:25:30.819528 7080 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 3b026c99d08842f5a61f0a658e08398a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:25:30.819805 7080 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 3b026c99d08842f5a61f0a658e08398a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:25:30.819878 7067 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 961f73cd6ac541f79ace3f8d78ff255b: attempting to promote NON_VOTER 1adca5af9a0644fcb8f274cbc0ae7292 to VOTER
I20260620 04:25:30.819906 7080 tablet_replica.cc:330] T 00000000000000000000000000000000 P 3b026c99d08842f5a61f0a658e08398a: stopping tablet replica
I20260620 04:25:30.820315 7067 consensus_queue.cc:237] T 00000000000000000000000000000000 P 961f73cd6ac541f79ace3f8d78ff255b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "961f73cd6ac541f79ace3f8d78ff255b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37461 } } peers { permanent_uuid: "1adca5af9a0644fcb8f274cbc0ae7292" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34531 } attrs { promote: false } } peers { permanent_uuid: "3b026c99d08842f5a61f0a658e08398a" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 45713 } attrs { promote: true } }
W20260620 04:25:30.822326 6495 consensus_peers.cc:487] T 00000000000000000000000000000000 P 961f73cd6ac541f79ace3f8d78ff255b -> Peer 3b026c99d08842f5a61f0a658e08398a (127.0.98.60:45713): Couldn't send request to peer 3b026c99d08842f5a61f0a658e08398a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 04:25:30.822379 7115 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1adca5af9a0644fcb8f274cbc0ae7292 [term 1 LEARNER]: Refusing update from remote peer 961f73cd6ac541f79ace3f8d78ff255b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260620 04:25:30.822775 7066 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 961f73cd6ac541f79ace3f8d78ff255b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1adca5af9a0644fcb8f274cbc0ae7292" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34531 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260620 04:25:30.825157 7066 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 961f73cd6ac541f79ace3f8d78ff255b [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 1adca5af9a0644fcb8f274cbc0ae7292 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "961f73cd6ac541f79ace3f8d78ff255b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37461 } } peers { permanent_uuid: "1adca5af9a0644fcb8f274cbc0ae7292" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34531 } attrs { promote: false } } peers { permanent_uuid: "3b026c99d08842f5a61f0a658e08398a" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 45713 } attrs { promote: true } } }
I20260620 04:25:30.826537 7066 sys_catalog.cc:455] T 00000000000000000000000000000000 P 961f73cd6ac541f79ace3f8d78ff255b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "961f73cd6ac541f79ace3f8d78ff255b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "961f73cd6ac541f79ace3f8d78ff255b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37461 } } peers { permanent_uuid: "1adca5af9a0644fcb8f274cbc0ae7292" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34531 } attrs { promote: false } } peers { permanent_uuid: "3b026c99d08842f5a61f0a658e08398a" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 45713 } attrs { promote: true } } }
I20260620 04:25:30.826637 7066 sys_catalog.cc:458] T 00000000000000000000000000000000 P 961f73cd6ac541f79ace3f8d78ff255b [sys.catalog]: This master's current role is: LEADER
I20260620 04:25:30.826738 7066 sys_catalog.cc:455] T 00000000000000000000000000000000 P 961f73cd6ac541f79ace3f8d78ff255b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "961f73cd6ac541f79ace3f8d78ff255b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "961f73cd6ac541f79ace3f8d78ff255b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37461 } } peers { permanent_uuid: "1adca5af9a0644fcb8f274cbc0ae7292" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34531 } attrs { promote: false } } peers { permanent_uuid: "3b026c99d08842f5a61f0a658e08398a" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 45713 } attrs { promote: true } } }
I20260620 04:25:30.826807 7066 sys_catalog.cc:458] T 00000000000000000000000000000000 P 961f73cd6ac541f79ace3f8d78ff255b [sys.catalog]: This master's current role is: LEADER
I20260620 04:25:30.827347 7115 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1adca5af9a0644fcb8f274cbc0ae7292 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 1adca5af9a0644fcb8f274cbc0ae7292 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "961f73cd6ac541f79ace3f8d78ff255b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37461 } } peers { permanent_uuid: "1adca5af9a0644fcb8f274cbc0ae7292" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34531 } attrs { promote: false } } peers { permanent_uuid: "3b026c99d08842f5a61f0a658e08398a" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 45713 } attrs { promote: true } } }
I20260620 04:25:30.828836 7254 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1adca5af9a0644fcb8f274cbc0ae7292 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "961f73cd6ac541f79ace3f8d78ff255b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "961f73cd6ac541f79ace3f8d78ff255b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37461 } } peers { permanent_uuid: "1adca5af9a0644fcb8f274cbc0ae7292" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34531 } attrs { promote: false } } peers { permanent_uuid: "3b026c99d08842f5a61f0a658e08398a" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 45713 } attrs { promote: true } } }
I20260620 04:25:30.828966 7254 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1adca5af9a0644fcb8f274cbc0ae7292 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 04:25:30.835656 7080 master.cc:552] Master@127.0.98.60:45713 shutdown complete.
I20260620 04:25:30.835713 7080 master_runner.cc:305] Clearing existing system tablet
I20260620 04:25:30.835935 7080 fs_manager.cc:359] Metadata directory not provided
I20260620 04:25:30.836011 7080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:25:30.837263 7080 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 04:25:30.838078 7257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:30.838299 7080 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:25:30.838423 7080 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 04:25:30.838454 7080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/master-2/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/master-2/wal
uuid: "3b026c99d08842f5a61f0a658e08398a"
format_stamp: "Formatted at 2026-06-20 04:25:30 on dist-test-slave-nk6d"
server_key: "344a51a282ec179295ac72a13be41482"
server_key_iv: "190bae46636527da7d75986f7e2a2221"
server_key_version: "encryptionkey@0"
I20260620 04:25:30.841041 7080 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 3b026c99d08842f5a61f0a658e08398a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:25:30.843355 7080 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 3b026c99d08842f5a61f0a658e08398a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 04:25:30.843449 7080 log.cc:1196] T 00000000000000000000000000000000 P 3b026c99d08842f5a61f0a658e08398a: Deleting WAL directory at /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260620 04:25:30.843745 7080 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 3b026c99d08842f5a61f0a658e08398a: Deleting consensus metadata
I20260620 04:25:30.843886 7080 master_runner.cc:315] Copying system tablet from 127.0.98.62:37461
I20260620 04:25:30.845003 7080 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 3b026c99d08842f5a61f0a658e08398a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:37461
I20260620 04:25:30.853515 7263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:30.845527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37461 (local address 127.0.0.1:44264)
0620 04:25:30.845810 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:30.845824 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:30.845946 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:30.846199 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:30.846202 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:30.846410 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:30.846515 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:30.846521 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:30.847546 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:30.847550 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:30.848571 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:30.848582 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:30.849125 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:30.849128 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:30.849219 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:30.849738 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:30.849751 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:30.850146 (+ 395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:25:30.851995 (+ 1849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:30.851999 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:30.852001 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:30.852230 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:30.852508 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:30.852511 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:30.852513 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:30.852601 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:30.852995 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:25:30.852998 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:25:30.853093 (+ 95us) client_negotiation.cc:769] Sending connection context
0620 04:25:30.853194 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 04:25:30.853326 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":78,"threads_started":1}
I20260620 04:25:30.853521 6547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:30.845634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44264 (local address 127.0.98.62:37461)
0620 04:25:30.845759 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:30.845763 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:30.845912 (+ 149us) server_negotiation.cc:407] Connection header received
0620 04:25:30.846027 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:30.846031 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:30.846081 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:30.846176 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:30.846623 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:30.847413 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:30.848696 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:30.849019 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:30.849123 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:30.850284 (+ 1161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:30.850288 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:30.850291 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:25:30.850317 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:30.851867 (+ 1550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:30.852326 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:30.852329 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:30.852331 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:30.852416 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:30.852691 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:30.852694 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:30.852696 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:30.852906 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:30.853000 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:30.853250 (+ 250us) server_negotiation.cc:299] Negotiation successful
0620 04:25:30.853368 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 04:25:30.854344 6528 tablet_copy_service.cc:140] P 961f73cd6ac541f79ace3f8d78ff255b: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 3b026c99d08842f5a61f0a658e08398a ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:44264)
I20260620 04:25:30.854456 6528 tablet_copy_service.cc:161] P 961f73cd6ac541f79ace3f8d78ff255b: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 3b026c99d08842f5a61f0a658e08398a at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:44264: session id = 3b026c99d08842f5a61f0a658e08398a-00000000000000000000000000000000
I20260620 04:25:30.855558 6528 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 961f73cd6ac541f79ace3f8d78ff255b: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 04:25:30.856817 7080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 04:25:30.859047 7080 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 3b026c99d08842f5a61f0a658e08398a: tablet copy: Starting download of 0 data blocks...
I20260620 04:25:30.859215 7080 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 3b026c99d08842f5a61f0a658e08398a: tablet copy: Starting download of 1 WAL segments...
I20260620 04:25:30.860634 7080 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 3b026c99d08842f5a61f0a658e08398a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 04:25:30.862217 6528 tablet_copy_service.cc:342] P 961f73cd6ac541f79ace3f8d78ff255b: Request end of tablet copy session 3b026c99d08842f5a61f0a658e08398a-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:44264
I20260620 04:25:30.862303 6528 tablet_copy_service.cc:433] P 961f73cd6ac541f79ace3f8d78ff255b: ending tablet copy session 3b026c99d08842f5a61f0a658e08398a-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 3b026c99d08842f5a61f0a658e08398a
I20260620 04:25:30.869210 7080 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:25:30.870738 7267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:25:30.870817 7264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:25:30.871557 7080 server_base.cc:1034] running on GCE node
W20260620 04:25:30.872110 7265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:25:30.872344 7080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:25:30.872660 7080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:25:30.873801 7080 hybrid_clock.cc:648] HybridClock initialized: now 1781929530873790 us; error 30 us; skew 500 ppm
Jun 20 04:25:30 dist-test-slave-nk6d krb5kdc[6463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929530, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:25:30.875360 7080 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260620 04:25:30.875892 7080 webserver.cc:460] Webserver started at http://127.0.98.60:43751/ using document root <none> and password file <none>
I20260620 04:25:30.876091 7080 fs_manager.cc:359] Metadata directory not provided
I20260620 04:25:30.876137 7080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:25:30.877460 7080 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 04:25:30.878504 7274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:30.878742 7080 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 04:25:30.878851 7080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/master-2/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/master-2/wal
uuid: "3b026c99d08842f5a61f0a658e08398a"
format_stamp: "Formatted at 2026-06-20 04:25:30 on dist-test-slave-nk6d"
server_key: "344a51a282ec179295ac72a13be41482"
server_key_iv: "190bae46636527da7d75986f7e2a2221"
server_key_version: "encryptionkey@0"
I20260620 04:25:30.878997 7080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:25:30.894508 7080 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:25:30.896632 7080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:25:30.901587 7080 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:45713
I20260620 04:25:30.902096 7326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:45713 every 8 connection(s)
I20260620 04:25:30.902483 7080 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/master-2/data/info.pb
I20260620 04:25:30.904394 7327 sys_catalog.cc:263] Verifying existing consensus state
I20260620 04:25:30.905580 7327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b026c99d08842f5a61f0a658e08398a: Bootstrap starting.
I20260620 04:25:30.916376 7327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b026c99d08842f5a61f0a658e08398a: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 04:25:30.916673 7327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b026c99d08842f5a61f0a658e08398a: Bootstrap complete.
I20260620 04:25:30.917241 7327 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b026c99d08842f5a61f0a658e08398a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "961f73cd6ac541f79ace3f8d78ff255b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37461 } } peers { permanent_uuid: "1adca5af9a0644fcb8f274cbc0ae7292" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34531 } attrs { promote: false } } peers { permanent_uuid: "3b026c99d08842f5a61f0a658e08398a" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 45713 } attrs { promote: true } }
I20260620 04:25:30.917382 7327 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b026c99d08842f5a61f0a658e08398a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3b026c99d08842f5a61f0a658e08398a, State: Initialized, Role: LEARNER
I20260620 04:25:30.917483 7327 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b026c99d08842f5a61f0a658e08398a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "961f73cd6ac541f79ace3f8d78ff255b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37461 } } peers { permanent_uuid: "1adca5af9a0644fcb8f274cbc0ae7292" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34531 } attrs { promote: false } } peers { permanent_uuid: "3b026c99d08842f5a61f0a658e08398a" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 45713 } attrs { promote: true } }
I20260620 04:25:30.917784 7333 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b026c99d08842f5a61f0a658e08398a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "961f73cd6ac541f79ace3f8d78ff255b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37461 } } peers { permanent_uuid: "1adca5af9a0644fcb8f274cbc0ae7292" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34531 } attrs { promote: false } } peers { permanent_uuid: "3b026c99d08842f5a61f0a658e08398a" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 45713 } attrs { promote: true } } }
I20260620 04:25:30.917893 7327 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b026c99d08842f5a61f0a658e08398a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:25:30.917981 7333 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b026c99d08842f5a61f0a658e08398a [sys.catalog]: This master's current role is: LEARNER
W20260620 04:25:30.918359 7252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:25:30.918618 7252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:25:30.918684 7252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260620 04:25:30.921280 7344 catalog_manager.cc:1259] Loaded cluster ID: 65e72dfb217a4b24b6c29af381fead6f
I20260620 04:25:30.921331 7344 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3b026c99d08842f5a61f0a658e08398a: loading cluster ID for follower catalog manager: success
I20260620 04:25:30.922945 7344 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3b026c99d08842f5a61f0a658e08398a: acquiring CA information for follower catalog manager: success
W20260620 04:25:30.923506 7252 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:25:30.923568 7252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:25:30.923651 7252 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 04:25:30.924203 7344 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3b026c99d08842f5a61f0a658e08398a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 04:25:30.929673 7252 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:42601
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=33081
--tserver_master_addrs=127.0.98.62:37461,127.0.98.61:34531,127.0.98.60:45713
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7252
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:25:30.930845 7252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:25:30.931980 7252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:25:30.941118 7349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:25:30.941184 7252 server_base.cc:1034] running on GCE node
W20260620 04:25:30.941147 7347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:25:30.941144 7346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:25:30.941694 7252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:25:30.942355 7252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:25:30.943504 7252 hybrid_clock.cc:648] HybridClock initialized: now 1781929530943473 us; error 45 us; skew 500 ppm
Jun 20 04:25:30 dist-test-slave-nk6d krb5kdc[6463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929530, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:25:30.946183 7252 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260620 04:25:30.946996 7252 webserver.cc:460] Webserver started at http://127.0.98.1:33081/ using document root <none> and password file <none>
I20260620 04:25:30.947513 7252 fs_manager.cc:359] Metadata directory not provided
I20260620 04:25:30.947590 7252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:25:30.949147 6833 heartbeater.cc:503] Master 127.0.98.62:37461 was elected leader, sending a full tablet report...
I20260620 04:25:30.951519 7252 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 04:25:30.953473 7356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:30.954658 7252 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 04:25:30.954763 7252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/ts-0/wal
uuid: "449f41f8c68044e29bb5d65d0f0652ef"
format_stamp: "Formatted at 2026-06-20 04:25:29 on dist-test-slave-nk6d"
server_key: "bcb2a96783a4c927e6532ff891a493fe"
server_key_iv: "4399317d5e3c1fafa96eb38a07b31998"
server_key_version: "encryptionkey@0"
I20260620 04:25:30.955283 7252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:25:30.966969 7252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:25:30.970124 7252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:25:30.970296 7252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:25:30.970858 7252 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:25:30.971663 7252 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:25:30.971716 7252 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:30.971768 7252 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:25:30.971797 7252 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:30.980342 7252 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42601
I20260620 04:25:30.980379 7469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42601 every 8 connection(s)
I20260620 04:25:30.981339 7252 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/ts-0/data/info.pb
I20260620 04:25:30.983757 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 7252
I20260620 04:25:30.983937 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 6704
Jun 20 04:25:30 dist-test-slave-nk6d krb5kdc[6463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929530, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 04:25:30.991583 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:38783
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:37461
--webserver_port=43693
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:37461,127.0.98.61:34531
--tserver_master_addrs=127.0.98.62:37461,127.0.98.61:34531,127.0.98.60:45713 with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 04:25:30 dist-test-slave-nk6d krb5kdc[6463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929530, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Jun 20 04:25:30 dist-test-slave-nk6d krb5kdc[6463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929530, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260620 04:25:30.996590 6547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:30.983013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44261 (local address 127.0.98.62:37461)
0620 04:25:30.983144 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:30.983148 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:30.984232 (+ 1084us) server_negotiation.cc:407] Connection header received
0620 04:25:30.985351 (+ 1119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:30.985355 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:30.985412 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:30.985460 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:30.986797 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:30.987579 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:30.989034 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:30.989338 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:30.989427 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:30.992576 (+ 3149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:30.992580 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:30.992583 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:25:30.992613 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:30.994571 (+ 1958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:30.995291 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:30.995295 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:30.995296 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:30.995380 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:30.995665 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:30.995668 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:30.995670 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:30.995871 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:30.995999 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:30.996320 (+ 321us) server_negotiation.cc:299] Negotiation successful
0620 04:25:30.996435 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 04:25:30.996848 7230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:30.983048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55961 (local address 127.0.98.61:34531)
0620 04:25:30.983148 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:30.983152 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:30.984911 (+ 1759us) server_negotiation.cc:407] Connection header received
0620 04:25:30.985315 (+ 404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:30.985318 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:30.985376 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:30.985451 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:30.986749 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:30.987578 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:30.988873 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:30.989124 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:30.989214 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:30.992570 (+ 3356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:30.992574 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:30.992577 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:25:30.992603 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:30.994202 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:30.994762 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:30.994766 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:30.994767 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:30.994847 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:30.995221 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:30.995225 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:30.995226 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:30.995419 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:30.995497 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:30.996574 (+ 1077us) server_negotiation.cc:299] Negotiation successful
0620 04:25:30.996691 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 04:25:30.997515 7474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:30.983509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37461 (local address 127.0.98.1:44261)
0620 04:25:30.984083 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:30.984136 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:30.985133 (+ 997us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:30.985630 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:30.985636 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:30.986121 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:30.986622 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:30.986634 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:30.987720 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:30.987723 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:30.988910 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:30.988919 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:30.989440 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:30.989443 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:30.989539 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:30.990128 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:30.990140 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:30.992392 (+ 2252us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:25:30.994689 (+ 2297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:30.994692 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:30.994694 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:30.995178 (+ 484us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:30.995486 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:30.995489 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:30.995491 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:30.995567 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:30.996054 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:25:30.996057 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:25:30.996141 (+ 84us) client_negotiation.cc:769] Sending connection context
0620 04:25:30.996736 (+ 595us) client_negotiation.cc:240] Negotiation successful
0620 04:25:30.996863 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"mutex_wait_us":19,"spinlock_wait_cycles":3840,"thread_start_us":127,"threads_started":1}
I20260620 04:25:30.997512 7476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:30.983509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34531 (local address 127.0.98.1:55961)
0620 04:25:30.984819 (+ 1310us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:30.984830 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:30.985137 (+ 307us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:30.985628 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:30.985635 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:30.986109 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:30.986620 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:30.986632 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:30.988035 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:30.988038 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:30.988765 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:30.988773 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:30.989258 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:30.989261 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:30.989441 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:30.990086 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:30.990105 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:30.992392 (+ 2287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:25:30.994307 (+ 1915us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:30.994313 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:30.994330 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:30.994662 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:30.994943 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:30.994946 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:30.994948 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:30.995125 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:30.995547 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:25:30.995553 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:25:30.995865 (+ 312us) client_negotiation.cc:769] Sending connection context
0620 04:25:30.996105 (+ 240us) client_negotiation.cc:240] Negotiation successful
0620 04:25:30.996473 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1214,"mutex_wait_us":56,"spinlock_wait_cycles":33280,"thread_start_us":669,"threads_started":1}
I20260620 04:25:30.998893 7472 heartbeater.cc:347] Connected to a master server at 127.0.98.61:34531
I20260620 04:25:30.999135 7472 heartbeater.cc:464] Registering TS with master...
I20260620 04:25:30.999140 7471 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37461
I20260620 04:25:30.999258 7471 heartbeater.cc:464] Registering TS with master...
I20260620 04:25:30.999629 7471 heartbeater.cc:511] Master 127.0.98.62:37461 requested a full tablet report, sending...
I20260620 04:25:30.999626 7472 heartbeater.cc:511] Master 127.0.98.61:34531 requested a full tablet report, sending...
I20260620 04:25:31.000949 6508 ts_manager.cc:194] Re-registered known tserver with Master: 449f41f8c68044e29bb5d65d0f0652ef (127.0.98.1:42601)
I20260620 04:25:31.001288 7108 ts_manager.cc:194] Registered new tserver with Master: 449f41f8c68044e29bb5d65d0f0652ef (127.0.98.1:42601)
I20260620 04:25:31.001560 6508 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:44261
I20260620 04:25:31.002573 7475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:30.984577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45713 (local address 127.0.98.1:49429)
0620 04:25:30.991251 (+ 6674us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:30.991263 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:30.991353 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:30.991641 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:30.991643 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:30.991817 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:30.991922 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:30.991929 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:30.993176 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:30.993180 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:30.993997 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:30.994008 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:30.994526 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:30.994530 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:30.994615 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:30.995008 (+ 393us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:30.995022 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:30.997583 (+ 2561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:25:30.999825 (+ 2242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:30.999828 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:30.999830 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:31.000083 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:31.000368 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:31.000371 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:31.000372 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:31.000480 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:31.002080 (+ 1600us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:25:31.002083 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:25:31.002182 (+ 99us) client_negotiation.cc:769] Sending connection context
0620 04:25:31.002266 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 04:25:31.002405 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6616,"thread_start_us":69,"threads_started":1}
I20260620 04:25:31.003053 7477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:30.985668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49429 (local address 127.0.98.60:45713)
0620 04:25:30.985935 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:30.985940 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:30.991449 (+ 5509us) server_negotiation.cc:407] Connection header received
0620 04:25:30.991492 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:30.991494 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:30.991538 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:30.991609 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:30.992069 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:30.993061 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:30.994125 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:30.994388 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:30.994490 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:30.997734 (+ 3244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:30.997739 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:30.997741 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:25:30.997767 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:30.999710 (+ 1943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:31.000187 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:31.000191 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:31.000193 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:31.000276 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:31.000572 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:31.000575 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:31.000577 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:31.001955 (+ 1378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:31.002041 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:31.002781 (+ 740us) server_negotiation.cc:299] Negotiation successful
0620 04:25:31.002905 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":84,"threads_started":1}
I20260620 04:25:31.003640 7470 heartbeater.cc:347] Connected to a master server at 127.0.98.60:45713
I20260620 04:25:31.003724 7470 heartbeater.cc:464] Registering TS with master...
I20260620 04:25:31.003921 7470 heartbeater.cc:511] Master 127.0.98.60:45713 requested a full tablet report, sending...
I20260620 04:25:31.005108 7291 ts_manager.cc:194] Registered new tserver with Master: 449f41f8c68044e29bb5d65d0f0652ef (127.0.98.1:42601)
W20260620 04:25:31.104121 7478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:25:31.104377 7478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:25:31.104432 7478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:25:31.108629 7478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:25:31.108690 7478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:25:31.108748 7478 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 04:25:31.113858 7478 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:38783
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=43693
--tserver_master_addrs=127.0.98.62:37461,127.0.98.61:34531,127.0.98.60:45713
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7478
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:25:31.114847 7478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:25:31.115774 7478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:25:31.125066 7484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:25:31.125066 7486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:25:31.125110 7483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:25:31.125295 7478 server_base.cc:1034] running on GCE node
I20260620 04:25:31.125706 7478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:25:31.126312 7478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:25:31.127466 7478 hybrid_clock.cc:648] HybridClock initialized: now 1781929531127447 us; error 33 us; skew 500 ppm
Jun 20 04:25:31 dist-test-slave-nk6d krb5kdc[6463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929531, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:25:31.130077 7478 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260620 04:25:31.130875 7478 webserver.cc:460] Webserver started at http://127.0.98.2:43693/ using document root <none> and password file <none>
I20260620 04:25:31.131407 7478 fs_manager.cc:359] Metadata directory not provided
I20260620 04:25:31.131484 7478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:25:31.135531 7478 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 04:25:31.137732 7493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:31.139106 7478 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 04:25:31.139221 7478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/ts-1/wal
uuid: "bde7e8cb28e54e59b808c1bb8439dbf1"
format_stamp: "Formatted at 2026-06-20 04:25:29 on dist-test-slave-nk6d"
server_key: "591dc49cdd34f572835a322fdf0c56c2"
server_key_iv: "294da74d391808e62783c48e6aa216f3"
server_key_version: "encryptionkey@0"
I20260620 04:25:31.139752 7478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:25:31.153502 6967 heartbeater.cc:503] Master 127.0.98.62:37461 was elected leader, sending a full tablet report...
I20260620 04:25:31.177094 7478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:25:31.180357 7478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:25:31.180541 7478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:25:31.181125 7478 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:25:31.181954 7478 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:25:31.181999 7478 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:31.182060 7478 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:25:31.182086 7478 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:31.191040 7478 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38783
I20260620 04:25:31.191063 7606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38783 every 8 connection(s)
I20260620 04:25:31.192072 7478 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/ts-1/data/info.pb
I20260620 04:25:31.199383 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 7478
I20260620 04:25:31.199548 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 6838
Jun 20 04:25:31 dist-test-slave-nk6d krb5kdc[6463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929531, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Jun 20 04:25:31 dist-test-slave-nk6d krb5kdc[6463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929531, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Jun 20 04:25:31 dist-test-slave-nk6d krb5kdc[6463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929531, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 04:25:31.206231 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:33941
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:37461
--webserver_port=34813
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:37461,127.0.98.61:34531
--tserver_master_addrs=127.0.98.62:37461,127.0.98.61:34531,127.0.98.60:45713 with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 04:25:31.206808 7065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:31.197545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45713 (local address 127.0.0.1:44132)
0620 04:25:31.197772 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:31.197785 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:31.197848 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:31.199195 (+ 1347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:31.199198 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:31.199215 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:25:31.199290 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:31.199295 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:31.202114 (+ 2819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:31.202117 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:31.204554 (+ 2437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:31.204567 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:31.206402 (+ 1835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:31.206406 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:31.206500 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:31.206525 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 04:25:31.206577 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 04:25:31.206660 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"mutex_wait_us":1230}
I20260620 04:25:31.207459 7614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:31.197975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44132 (local address 127.0.98.60:45713)
0620 04:25:31.198948 (+ 973us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:31.198952 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:31.198964 (+ 12us) server_negotiation.cc:407] Connection header received
0620 04:25:31.199005 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:31.199007 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:31.199063 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:31.199386 (+ 323us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:25:31.199474 (+ 88us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:31.201993 (+ 2519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:31.205623 (+ 3630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:31.206258 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:31.207008 (+ 750us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:31.207201 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:31.207238 (+ 37us) server_negotiation.cc:299] Negotiation successful
0620 04:25:31.207307 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":905,"thread_start_us":83,"threads_started":1}
I20260620 04:25:31.208045 7301 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 3b026c99d08842f5a61f0a658e08398a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260620 04:25:31.208606 7333 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b026c99d08842f5a61f0a658e08398a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 961f73cd6ac541f79ace3f8d78ff255b. Latest consensus state: current_term: 1 leader_uuid: "961f73cd6ac541f79ace3f8d78ff255b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "961f73cd6ac541f79ace3f8d78ff255b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37461 } } peers { permanent_uuid: "1adca5af9a0644fcb8f274cbc0ae7292" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34531 } attrs { promote: false } } peers { permanent_uuid: "3b026c99d08842f5a61f0a658e08398a" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 45713 } attrs { promote: true } } }
I20260620 04:25:31.208742 7333 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b026c99d08842f5a61f0a658e08398a [sys.catalog]: This master's current role is: LEARNER
I20260620 04:25:31.210666 7230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:31.193795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50641 (local address 127.0.98.61:34531)
0620 04:25:31.193925 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:31.193928 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:31.195275 (+ 1347us) server_negotiation.cc:407] Connection header received
0620 04:25:31.196117 (+ 842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:31.196119 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:31.196166 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:31.196255 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:31.197701 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:31.198351 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:31.199951 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:31.200292 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:31.200382 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:31.205626 (+ 5244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:31.205629 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:31.205632 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:25:31.205659 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:31.207574 (+ 1915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:31.208187 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:31.208191 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:31.208193 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:31.208274 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:31.208693 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:31.208697 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:31.208699 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:31.208923 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:31.209022 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:31.210395 (+ 1373us) server_negotiation.cc:299] Negotiation successful
0620 04:25:31.210517 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 04:25:31.210965 6547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:31.193792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53049 (local address 127.0.98.62:37461)
0620 04:25:31.193919 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:31.193923 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:31.194959 (+ 1036us) server_negotiation.cc:407] Connection header received
0620 04:25:31.196056 (+ 1097us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:31.196060 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:31.196116 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:31.196182 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:31.197702 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:31.198519 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:31.202680 (+ 4161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:31.203123 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:31.203219 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:31.205299 (+ 2080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:31.205303 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:31.205306 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:25:31.205330 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:31.209134 (+ 3804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:31.209792 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:31.209795 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:31.209797 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:31.209885 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:31.210158 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:31.210161 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:31.210163 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:31.210351 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:31.210419 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:31.210717 (+ 298us) server_negotiation.cc:299] Negotiation successful
0620 04:25:31.210818 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":37,"server-negotiator.queue_time_us":53}
I20260620 04:25:31.211261 7613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:31.195320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45713 (local address 127.0.98.2:33813)
0620 04:25:31.195565 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:31.195577 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:31.195803 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:31.196441 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:31.196444 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:31.196880 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:31.197552 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:31.197563 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:31.198650 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:31.198654 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:31.199672 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:31.199687 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:31.201030 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:31.201033 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:31.201160 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:31.201581 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:31.201600 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:31.205108 (+ 3508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:25:31.209206 (+ 4098us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:31.209208 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:31.209210 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:31.209366 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:31.209635 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:31.209638 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:31.209640 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:31.209722 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:31.210137 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:25:31.210140 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:25:31.210226 (+ 86us) client_negotiation.cc:769] Sending connection context
0620 04:25:31.210312 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 04:25:31.210454 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"mutex_wait_us":120,"spinlock_wait_cycles":4352,"thread_start_us":81,"threads_started":1}
I20260620 04:25:31.211665 7612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:31.194285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34531 (local address 127.0.98.2:50641)
0620 04:25:31.195175 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:31.195187 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:31.195796 (+ 609us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:31.196378 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:31.196387 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:31.196879 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:31.197542 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:31.197563 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:31.198484 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:31.198488 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:31.199849 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:31.199854 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:31.200721 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:31.200724 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:31.200920 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:31.201681 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:31.201696 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:31.205481 (+ 3785us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:25:31.207680 (+ 2199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:31.207686 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:31.207704 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:31.208075 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:31.208383 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:31.208386 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:31.208388 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:31.208584 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:31.209040 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:25:31.209047 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:25:31.209372 (+ 325us) client_negotiation.cc:769] Sending connection context
0620 04:25:31.209573 (+ 201us) client_negotiation.cc:240] Negotiation successful
0620 04:25:31.209902 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":806,"mutex_wait_us":372,"spinlock_wait_cycles":58496,"thread_start_us":626,"threads_started":1}
I20260620 04:25:31.211743 7611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:31.194285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37461 (local address 127.0.98.2:53049)
0620 04:25:31.194796 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:31.194836 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:31.195796 (+ 960us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:31.196379 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:31.196388 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:31.196880 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:31.197549 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:31.197563 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:31.198650 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:31.198654 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:31.202561 (+ 3907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:31.202571 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:31.203250 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:31.203254 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:31.203360 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:31.203830 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:31.203845 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:31.205168 (+ 1323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:25:31.209475 (+ 4307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:31.209478 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:31.209480 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:31.209691 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:31.209983 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:31.209985 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:31.209988 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:31.210065 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:31.210468 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:25:31.210471 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:25:31.210555 (+ 84us) client_negotiation.cc:769] Sending connection context
0620 04:25:31.211127 (+ 572us) client_negotiation.cc:240] Negotiation successful
0620 04:25:31.211256 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"mutex_wait_us":133,"spinlock_wait_cycles":23936,"thread_start_us":136,"threads_started":1}
I20260620 04:25:31.212102 7477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:31.195465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33813 (local address 127.0.98.60:45713)
0620 04:25:31.195587 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:31.195590 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:31.195630 (+ 40us) server_negotiation.cc:407] Connection header received
0620 04:25:31.196294 (+ 664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:31.196297 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:31.196347 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:31.196419 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:31.197702 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:31.198534 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:31.200123 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:31.200455 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:31.200539 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:31.205266 (+ 4727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:31.205270 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:31.205273 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:25:31.205299 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:31.208304 (+ 3005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:31.209455 (+ 1151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:31.209459 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:31.209461 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:31.209539 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:31.209812 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:31.209815 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:31.209817 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:31.209999 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:31.210099 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:31.211838 (+ 1739us) server_negotiation.cc:299] Negotiation successful
0620 04:25:31.211948 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 04:25:31.212770 7609 heartbeater.cc:347] Connected to a master server at 127.0.98.61:34531
I20260620 04:25:31.212852 7607 heartbeater.cc:347] Connected to a master server at 127.0.98.60:45713
I20260620 04:25:31.212998 7607 heartbeater.cc:464] Registering TS with master...
I20260620 04:25:31.213022 7608 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37461
I20260620 04:25:31.212997 7609 heartbeater.cc:464] Registering TS with master...
I20260620 04:25:31.213107 7608 heartbeater.cc:464] Registering TS with master...
I20260620 04:25:31.213586 7609 heartbeater.cc:511] Master 127.0.98.61:34531 requested a full tablet report, sending...
I20260620 04:25:31.213586 7607 heartbeater.cc:511] Master 127.0.98.60:45713 requested a full tablet report, sending...
I20260620 04:25:31.213627 7608 heartbeater.cc:511] Master 127.0.98.62:37461 requested a full tablet report, sending...
I20260620 04:25:31.214607 6508 ts_manager.cc:194] Re-registered known tserver with Master: bde7e8cb28e54e59b808c1bb8439dbf1 (127.0.98.2:38783)
I20260620 04:25:31.214701 7108 ts_manager.cc:194] Registered new tserver with Master: bde7e8cb28e54e59b808c1bb8439dbf1 (127.0.98.2:38783)
I20260620 04:25:31.214735 7291 ts_manager.cc:194] Registered new tserver with Master: bde7e8cb28e54e59b808c1bb8439dbf1 (127.0.98.2:38783)
I20260620 04:25:31.215214 6508 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53049
W20260620 04:25:31.318781 7615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:25:31.319048 7615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:25:31.319106 7615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:25:31.323276 7615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:25:31.323333 7615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:25:31.323390 7615 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 04:25:31.328639 7615 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:33941
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=34813
--tserver_master_addrs=127.0.98.62:37461,127.0.98.61:34531,127.0.98.60:45713
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7615
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:25:31.329636 7615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:25:31.330605 7615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:25:31.339705 7620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:25:31.339871 7623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:25:31.339951 7615 server_base.cc:1034] running on GCE node
W20260620 04:25:31.340107 7621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:25:31.340555 7615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:25:31.341150 7615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:25:31.342316 7615 hybrid_clock.cc:648] HybridClock initialized: now 1781929531342285 us; error 44 us; skew 500 ppm
Jun 20 04:25:31 dist-test-slave-nk6d krb5kdc[6463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929531, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:25:31.345032 7615 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260620 04:25:31.345835 7615 webserver.cc:460] Webserver started at http://127.0.98.3:34813/ using document root <none> and password file <none>
I20260620 04:25:31.346321 7615 fs_manager.cc:359] Metadata directory not provided
I20260620 04:25:31.346385 7615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:25:31.350082 7615 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:25:31.351980 7630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:31.353165 7615 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:25:31.353258 7615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/ts-2/wal
uuid: "66d7beaf7d8a415f8e5e1b457f1ec504"
format_stamp: "Formatted at 2026-06-20 04:25:30 on dist-test-slave-nk6d"
server_key: "50339e8927dba87a693b0fe89e778b0b"
server_key_iv: "a1ecc2c08068a7f064c9b09db8aacc60"
server_key_version: "encryptionkey@0"
I20260620 04:25:31.353760 7615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:25:31.367858 7615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:25:31.370896 7615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:25:31.371044 7615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:25:31.371570 7615 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:25:31.372408 7615 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:25:31.372466 7615 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:31.372504 7615 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:25:31.372524 7615 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:31.380676 7615 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33941
I20260620 04:25:31.380723 7743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33941 every 8 connection(s)
I20260620 04:25:31.381634 7615 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/ts-2/data/info.pb
I20260620 04:25:31.383240 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 7615
Jun 20 04:25:31 dist-test-slave-nk6d krb5kdc[6463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929531, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Jun 20 04:25:31 dist-test-slave-nk6d krb5kdc[6463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929531, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Jun 20 04:25:31 dist-test-slave-nk6d krb5kdc[6463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929531, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260620 04:25:31.396865 7757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:31.385356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49064 (local address 127.0.98.61:34531)
0620 04:25:31.385949 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:31.385953 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:31.387270 (+ 1317us) server_negotiation.cc:407] Connection header received
0620 04:25:31.387308 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:31.387311 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:31.387351 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:31.387419 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:31.388256 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:31.389015 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:31.390173 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:31.390485 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:31.390571 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:31.392560 (+ 1989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:31.392564 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:31.392567 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:25:31.392590 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:31.394997 (+ 2407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:31.395320 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:31.395323 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:31.395325 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:31.395401 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:31.395666 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:31.395669 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:31.395671 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:31.395872 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:31.396027 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:31.396528 (+ 501us) server_negotiation.cc:299] Negotiation successful
0620 04:25:31.396690 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":69,"threads_started":1}
I20260620 04:25:31.399191 7614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:31.385690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43129 (local address 127.0.98.60:45713)
0620 04:25:31.385786 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:31.385790 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:31.385971 (+ 181us) server_negotiation.cc:407] Connection header received
0620 04:25:31.386125 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:31.386128 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:31.386175 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:31.386221 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:31.388172 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:31.388963 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:31.391551 (+ 2588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:31.391862 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:31.391977 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:31.394326 (+ 2349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:31.394330 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:31.394332 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:25:31.394354 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:31.396450 (+ 2096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:31.397063 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:31.397066 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:31.397068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:31.397143 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:31.397970 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:31.397973 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:31.397975 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:31.398171 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:31.398261 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:31.398888 (+ 627us) server_negotiation.cc:299] Negotiation successful
0620 04:25:31.399040 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 04:25:31.400554 7230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:31.383327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47787 (local address 127.0.98.61:34531)
0620 04:25:31.383451 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:31.383455 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:31.384814 (+ 1359us) server_negotiation.cc:407] Connection header received
0620 04:25:31.385960 (+ 1146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:31.385964 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:31.386014 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:31.386065 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:31.390225 (+ 4160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:31.390881 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:31.393943 (+ 3062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:31.394255 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:31.394352 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:31.397299 (+ 2947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:31.397303 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:31.397305 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:25:31.397327 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:31.398884 (+ 1557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:31.399340 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:31.399343 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:31.399345 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:31.399419 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:31.399710 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:31.399714 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:31.399715 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:31.399892 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:31.399983 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:31.400306 (+ 323us) server_negotiation.cc:299] Negotiation successful
0620 04:25:31.400400 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 04:25:31.400658 6547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:31.383518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35543 (local address 127.0.98.62:37461)
0620 04:25:31.384757 (+ 1239us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:31.384761 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:31.385748 (+ 987us) server_negotiation.cc:407] Connection header received
0620 04:25:31.385930 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:31.385934 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:31.385986 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:31.386041 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:31.388007 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:31.388740 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:31.390401 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:31.390706 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:31.390773 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:31.393608 (+ 2835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:31.393612 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:31.393615 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:25:31.393639 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:31.396357 (+ 2718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:31.396949 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:31.396953 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:31.396954 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:31.397032 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:31.398753 (+ 1721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:31.398757 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:31.398759 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:31.398981 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:31.399076 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:31.400427 (+ 1351us) server_negotiation.cc:299] Negotiation successful
0620 04:25:31.400527 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1155}
I20260620 04:25:31.401255 7477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:31.385243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44142 (local address 127.0.98.60:45713)
0620 04:25:31.385551 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:31.385555 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:31.386878 (+ 1323us) server_negotiation.cc:407] Connection header received
0620 04:25:31.386996 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:31.386999 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:31.387043 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:31.387127 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:31.390567 (+ 3440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:31.391730 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:31.395044 (+ 3314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:31.395364 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:31.395478 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:31.398019 (+ 2541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:31.398023 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:31.398025 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:25:31.398045 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:31.399572 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:31.400075 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:31.400078 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:31.400080 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:31.400155 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:31.400418 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:31.400433 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:31.400435 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:31.400613 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:31.400719 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:31.401022 (+ 303us) server_negotiation.cc:299] Negotiation successful
0620 04:25:31.401128 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"spinlock_wait_cycles":1792}
I20260620 04:25:31.401345 7758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:31.385082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37461 (local address 127.0.98.3:35543)
0620 04:25:31.385655 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:31.385668 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:31.385816 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:31.386439 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:31.386442 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:31.387056 (+ 614us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:31.387839 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:31.387859 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:31.388903 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:31.388906 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:31.390281 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:31.390291 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:31.390944 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:31.390949 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:31.391157 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:31.391629 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:31.391647 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:31.393418 (+ 1771us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:25:31.396480 (+ 3062us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:31.396486 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:31.396501 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:31.396850 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:31.397572 (+ 722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:31.397574 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:31.397576 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:31.398648 (+ 1072us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:31.399141 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:25:31.399144 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:25:31.399227 (+ 83us) client_negotiation.cc:769] Sending connection context
0620 04:25:31.401007 (+ 1780us) client_negotiation.cc:240] Negotiation successful
0620 04:25:31.401122 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"mutex_wait_us":94,"spinlock_wait_cycles":13184,"thread_start_us":80,"threads_started":1}
I20260620 04:25:31.401626 7755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:31.385386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45713 (local address 127.0.98.3:43129)
0620 04:25:31.385907 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:31.385919 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:31.386031 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:31.386434 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:31.386441 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:31.387146 (+ 705us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:31.388048 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:31.388057 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:31.389093 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:31.389096 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:31.391450 (+ 2354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:31.391460 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:31.392094 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:31.392098 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:31.392192 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:31.392784 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:31.392797 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:31.394198 (+ 1401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:25:31.396679 (+ 2481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:31.396682 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:31.396683 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:31.396962 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:31.397466 (+ 504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:31.397468 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:31.397471 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:31.397860 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:31.398312 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:25:31.398318 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:25:31.398618 (+ 300us) client_negotiation.cc:769] Sending connection context
0620 04:25:31.399513 (+ 895us) client_negotiation.cc:240] Negotiation successful
0620 04:25:31.399845 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"mutex_wait_us":1678,"thread_start_us":97,"threads_started":1}
I20260620 04:25:31.401729 7752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:31.383768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34531 (local address 127.0.98.3:47787)
0620 04:25:31.384666 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:31.384719 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:31.385715 (+ 996us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:31.386432 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:31.386441 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:31.387505 (+ 1064us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:31.387840 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:31.387859 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:31.391004 (+ 3145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:31.391007 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:31.392649 (+ 1642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:31.392658 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:31.394490 (+ 1832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:31.394494 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:31.394583 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:31.395605 (+ 1022us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:31.395619 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:31.397175 (+ 1556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:25:31.399024 (+ 1849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:31.399027 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:31.399029 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:31.399237 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:31.399534 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:31.399537 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:31.399538 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:31.399614 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:31.400036 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:25:31.400050 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:25:31.400134 (+ 84us) client_negotiation.cc:769] Sending connection context
0620 04:25:31.400728 (+ 594us) client_negotiation.cc:240] Negotiation successful
0620 04:25:31.400851 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":701,"mutex_wait_us":41,"thread_start_us":563,"threads_started":1}
I20260620 04:25:31.402616 7746 heartbeater.cc:347] Connected to a master server at 127.0.98.61:34531
I20260620 04:25:31.402668 7744 heartbeater.cc:347] Connected to a master server at 127.0.98.60:45713
I20260620 04:25:31.402804 7745 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37461
I20260620 04:25:31.402870 7745 heartbeater.cc:464] Registering TS with master...
I20260620 04:25:31.402858 7744 heartbeater.cc:464] Registering TS with master...
I20260620 04:25:31.402856 7746 heartbeater.cc:464] Registering TS with master...
I20260620 04:25:31.403398 7746 heartbeater.cc:511] Master 127.0.98.61:34531 requested a full tablet report, sending...
I20260620 04:25:31.403410 7744 heartbeater.cc:511] Master 127.0.98.60:45713 requested a full tablet report, sending...
I20260620 04:25:31.403843 7754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:31.384879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44270 (local address 127.0.98.62:37461)
0620 04:25:31.386348 (+ 1469us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:31.386352 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:31.386364 (+ 12us) server_negotiation.cc:407] Connection header received
0620 04:25:31.386393 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:31.386396 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:31.386430 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:31.386502 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:31.387722 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:31.389475 (+ 1753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:31.391045 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:31.391381 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:31.391470 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:31.394320 (+ 2850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:31.394324 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:31.394326 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:25:31.394350 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:31.399881 (+ 5531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:31.402003 (+ 2122us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:31.402006 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:31.402008 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:31.402094 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:31.402984 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:31.402987 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:31.402988 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:31.403183 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:31.403300 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:31.403587 (+ 287us) server_negotiation.cc:299] Negotiation successful
0620 04:25:31.403693 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1405,"thread_start_us":91,"threads_started":1}
I20260620 04:25:31.403398 7745 heartbeater.cc:511] Master 127.0.98.62:37461 requested a full tablet report, sending...
I20260620 04:25:31.404486 7108 ts_manager.cc:194] Registered new tserver with Master: 66d7beaf7d8a415f8e5e1b457f1ec504 (127.0.98.3:33941)
I20260620 04:25:31.404486 7291 ts_manager.cc:194] Registered new tserver with Master: 66d7beaf7d8a415f8e5e1b457f1ec504 (127.0.98.3:33941)
I20260620 04:25:31.404549 6507 ts_manager.cc:194] Re-registered known tserver with Master: 66d7beaf7d8a415f8e5e1b457f1ec504 (127.0.98.3:33941)
I20260620 04:25:31.405467 6507 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:35543
I20260620 04:25:31.406574 6507 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44270:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 04:25:31.409446 6507 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 04:25:31.411820 7255 sys_catalog.cc:455] T 00000000000000000000000000000000 P 961f73cd6ac541f79ace3f8d78ff255b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "961f73cd6ac541f79ace3f8d78ff255b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "961f73cd6ac541f79ace3f8d78ff255b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37461 } } peers { permanent_uuid: "1adca5af9a0644fcb8f274cbc0ae7292" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34531 } attrs { promote: false } } peers { permanent_uuid: "3b026c99d08842f5a61f0a658e08398a" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 45713 } attrs { promote: true } } }
I20260620 04:25:31.411948 7255 sys_catalog.cc:458] T 00000000000000000000000000000000 P 961f73cd6ac541f79ace3f8d78ff255b [sys.catalog]: This master's current role is: LEADER
I20260620 04:25:31.415230 7761 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 961f73cd6ac541f79ace3f8d78ff255b: attempting to promote NON_VOTER 3b026c99d08842f5a61f0a658e08398a to VOTER
I20260620 04:25:31.415586 7761 consensus_queue.cc:237] T 00000000000000000000000000000000 P 961f73cd6ac541f79ace3f8d78ff255b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "961f73cd6ac541f79ace3f8d78ff255b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37461 } } peers { permanent_uuid: "1adca5af9a0644fcb8f274cbc0ae7292" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34531 } attrs { promote: false } } peers { permanent_uuid: "3b026c99d08842f5a61f0a658e08398a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45713 } attrs { promote: false } }
I20260620 04:25:31.416535 7301 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3b026c99d08842f5a61f0a658e08398a [term 1 LEARNER]: Refusing update from remote peer 961f73cd6ac541f79ace3f8d78ff255b: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260620 04:25:31.416946 7762 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 961f73cd6ac541f79ace3f8d78ff255b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b026c99d08842f5a61f0a658e08398a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260620 04:25:31.418396 7763 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 961f73cd6ac541f79ace3f8d78ff255b [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 3b026c99d08842f5a61f0a658e08398a (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "961f73cd6ac541f79ace3f8d78ff255b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37461 } } peers { permanent_uuid: "1adca5af9a0644fcb8f274cbc0ae7292" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34531 } attrs { promote: false } } peers { permanent_uuid: "3b026c99d08842f5a61f0a658e08398a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45713 } attrs { promote: false } } }
I20260620 04:25:31.418794 7301 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3b026c99d08842f5a61f0a658e08398a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 3b026c99d08842f5a61f0a658e08398a (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "961f73cd6ac541f79ace3f8d78ff255b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37461 } } peers { permanent_uuid: "1adca5af9a0644fcb8f274cbc0ae7292" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34531 } attrs { promote: false } } peers { permanent_uuid: "3b026c99d08842f5a61f0a658e08398a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45713 } attrs { promote: false } } }
I20260620 04:25:31.419227 7115 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1adca5af9a0644fcb8f274cbc0ae7292 [term 1 FOLLOWER]: Refusing update from remote peer 961f73cd6ac541f79ace3f8d78ff255b: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260620 04:25:31.419560 7762 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 961f73cd6ac541f79ace3f8d78ff255b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1adca5af9a0644fcb8f274cbc0ae7292" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34531 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260620 04:25:31.420384 7115 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1adca5af9a0644fcb8f274cbc0ae7292 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 3b026c99d08842f5a61f0a658e08398a (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "961f73cd6ac541f79ace3f8d78ff255b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37461 } } peers { permanent_uuid: "1adca5af9a0644fcb8f274cbc0ae7292" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34531 } attrs { promote: false } } peers { permanent_uuid: "3b026c99d08842f5a61f0a658e08398a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45713 } attrs { promote: false } } }
I20260620 04:25:31.419879 7763 sys_catalog.cc:455] T 00000000000000000000000000000000 P 961f73cd6ac541f79ace3f8d78ff255b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "961f73cd6ac541f79ace3f8d78ff255b" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "961f73cd6ac541f79ace3f8d78ff255b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37461 } } peers { permanent_uuid: "1adca5af9a0644fcb8f274cbc0ae7292" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34531 } attrs { promote: false } } peers { permanent_uuid: "3b026c99d08842f5a61f0a658e08398a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45713 } attrs { promote: false } } }
I20260620 04:25:31.420753 7763 sys_catalog.cc:458] T 00000000000000000000000000000000 P 961f73cd6ac541f79ace3f8d78ff255b [sys.catalog]: This master's current role is: LEADER
I20260620 04:25:31.420718 7333 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b026c99d08842f5a61f0a658e08398a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "961f73cd6ac541f79ace3f8d78ff255b" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "961f73cd6ac541f79ace3f8d78ff255b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37461 } } peers { permanent_uuid: "1adca5af9a0644fcb8f274cbc0ae7292" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34531 } attrs { promote: false } } peers { permanent_uuid: "3b026c99d08842f5a61f0a658e08398a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45713 } attrs { promote: false } } }
I20260620 04:25:31.420739 7761 sys_catalog.cc:455] T 00000000000000000000000000000000 P 961f73cd6ac541f79ace3f8d78ff255b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "961f73cd6ac541f79ace3f8d78ff255b" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "961f73cd6ac541f79ace3f8d78ff255b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37461 } } peers { permanent_uuid: "1adca5af9a0644fcb8f274cbc0ae7292" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34531 } attrs { promote: false } } peers { permanent_uuid: "3b026c99d08842f5a61f0a658e08398a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45713 } attrs { promote: false } } }
I20260620 04:25:31.420817 7333 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b026c99d08842f5a61f0a658e08398a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 04:25:31.420835 7761 sys_catalog.cc:458] T 00000000000000000000000000000000 P 961f73cd6ac541f79ace3f8d78ff255b [sys.catalog]: This master's current role is: LEADER
I20260620 04:25:31.421741 7771 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1adca5af9a0644fcb8f274cbc0ae7292 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "961f73cd6ac541f79ace3f8d78ff255b" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "961f73cd6ac541f79ace3f8d78ff255b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37461 } } peers { permanent_uuid: "1adca5af9a0644fcb8f274cbc0ae7292" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34531 } attrs { promote: false } } peers { permanent_uuid: "3b026c99d08842f5a61f0a658e08398a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45713 } attrs { promote: false } } }
I20260620 04:25:31.421864 7771 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1adca5af9a0644fcb8f274cbc0ae7292 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 04:25:31.422982 7763 sys_catalog.cc:455] T 00000000000000000000000000000000 P 961f73cd6ac541f79ace3f8d78ff255b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "961f73cd6ac541f79ace3f8d78ff255b" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "961f73cd6ac541f79ace3f8d78ff255b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37461 } } peers { permanent_uuid: "1adca5af9a0644fcb8f274cbc0ae7292" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34531 } attrs { promote: false } } peers { permanent_uuid: "3b026c99d08842f5a61f0a658e08398a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45713 } attrs { promote: false } } }
I20260620 04:25:31.423085 7763 sys_catalog.cc:458] T 00000000000000000000000000000000 P 961f73cd6ac541f79ace3f8d78ff255b [sys.catalog]: This master's current role is: LEADER
I20260620 04:25:31.430044 7065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:31.424404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33941 (local address 127.0.0.1:60584)
0620 04:25:31.425228 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:31.425244 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:31.425322 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:31.426193 (+ 871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:31.426196 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:31.426213 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:25:31.426313 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:31.426319 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:31.427392 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:31.427395 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:31.428597 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:31.428607 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:31.429624 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:31.429627 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:31.429727 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:31.429764 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 04:25:31.429813 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 04:25:31.429876 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":757,"mutex_wait_us":52}
I20260620 04:25:31.430047 7772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:31.424611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42601 (local address 127.0.0.1:59298)
0620 04:25:31.425694 (+ 1083us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:31.425708 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:31.425789 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:31.426226 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:31.426229 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:31.426248 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:25:31.426342 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:31.426347 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:31.427618 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:31.427621 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:31.428812 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:31.428821 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:31.429606 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:31.429609 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:31.429701 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:31.429764 (+ 63us) client_negotiation.cc:769] Sending connection context
0620 04:25:31.429831 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 04:25:31.429879 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1022,"mutex_wait_us":66,"thread_start_us":456,"threads_started":1}
I20260620 04:25:31.430087 7776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:31.425426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59298 (local address 127.0.98.1:42601)
0620 04:25:31.425765 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:31.425776 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:31.425799 (+ 23us) server_negotiation.cc:407] Connection header received
0620 04:25:31.425884 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:31.425889 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:31.426108 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:31.426237 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:25:31.426423 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:31.427514 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:31.428927 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:31.429473 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:31.429592 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:31.429740 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:31.429858 (+ 118us) server_negotiation.cc:299] Negotiation successful
0620 04:25:31.429945 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":89,"threads_started":1}
I20260620 04:25:31.430787 7774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:31.424554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60584 (local address 127.0.98.3:33941)
0620 04:25:31.425791 (+ 1237us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:31.425800 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:31.425818 (+ 18us) server_negotiation.cc:407] Connection header received
0620 04:25:31.425881 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:31.425886 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:31.426066 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:31.426206 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:25:31.426431 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:31.427306 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:31.428709 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:31.429513 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:31.430477 (+ 964us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:31.430579 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:31.430617 (+ 38us) server_negotiation.cc:299] Negotiation successful
0620 04:25:31.430680 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1157,"thread_start_us":512,"threads_started":1}
I20260620 04:25:31.431779 7775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:31.425151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36522 (local address 127.0.98.2:38783)
0620 04:25:31.425999 (+ 848us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:31.426009 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:31.426497 (+ 488us) server_negotiation.cc:407] Connection header received
0620 04:25:31.426604 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:31.426609 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:31.426814 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:31.426950 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:25:31.427170 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:31.428151 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:31.429529 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:31.430277 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:31.430368 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:31.430491 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:31.431560 (+ 1069us) server_negotiation.cc:299] Negotiation successful
0620 04:25:31.431647 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":761,"thread_start_us":98,"threads_started":1}
I20260620 04:25:31.431948 7773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:31.424860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38783 (local address 127.0.0.1:36522)
0620 04:25:31.426417 (+ 1557us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:31.426428 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:31.426511 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:31.426940 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:31.426942 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:31.426959 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:25:31.427058 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:31.427063 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:31.428262 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:31.428266 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:31.429398 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:31.429408 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:31.430550 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:31.430554 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:31.430636 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:31.430658 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 04:25:31.430720 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 04:25:31.430782 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1500,"mutex_wait_us":2,"thread_start_us":97,"threads_started":1}
I20260620 04:25:31.433182 7404 tablet_service.cc:1467] Processing CreateTablet for tablet df85ee7a57664f14b5ad6e1b2bd8ded4 (DEFAULT_TABLE table=test-table [id=8494abfc67f3480bafd0dc30317150da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:25:31.433248 7678 tablet_service.cc:1467] Processing CreateTablet for tablet df85ee7a57664f14b5ad6e1b2bd8ded4 (DEFAULT_TABLE table=test-table [id=8494abfc67f3480bafd0dc30317150da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:25:31.434037 7404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df85ee7a57664f14b5ad6e1b2bd8ded4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:25:31.434226 7678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df85ee7a57664f14b5ad6e1b2bd8ded4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:25:31.434443 7541 tablet_service.cc:1467] Processing CreateTablet for tablet df85ee7a57664f14b5ad6e1b2bd8ded4 (DEFAULT_TABLE table=test-table [id=8494abfc67f3480bafd0dc30317150da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:25:31.435333 7541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df85ee7a57664f14b5ad6e1b2bd8ded4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:25:31.441404 7777 tablet_bootstrap.cc:492] T df85ee7a57664f14b5ad6e1b2bd8ded4 P 449f41f8c68044e29bb5d65d0f0652ef: Bootstrap starting.
I20260620 04:25:31.442332 7778 tablet_bootstrap.cc:492] T df85ee7a57664f14b5ad6e1b2bd8ded4 P 66d7beaf7d8a415f8e5e1b457f1ec504: Bootstrap starting.
I20260620 04:25:31.443627 7779 tablet_bootstrap.cc:492] T df85ee7a57664f14b5ad6e1b2bd8ded4 P bde7e8cb28e54e59b808c1bb8439dbf1: Bootstrap starting.
I20260620 04:25:31.443981 7777 tablet_bootstrap.cc:654] T df85ee7a57664f14b5ad6e1b2bd8ded4 P 449f41f8c68044e29bb5d65d0f0652ef: Neither blocks nor log segments found. Creating new log.
I20260620 04:25:31.444475 7778 tablet_bootstrap.cc:654] T df85ee7a57664f14b5ad6e1b2bd8ded4 P 66d7beaf7d8a415f8e5e1b457f1ec504: Neither blocks nor log segments found. Creating new log.
I20260620 04:25:31.444839 7777 log.cc:824] T df85ee7a57664f14b5ad6e1b2bd8ded4 P 449f41f8c68044e29bb5d65d0f0652ef: Log is configured to *not* fsync() on all Append() calls
I20260620 04:25:31.445215 7778 log.cc:824] T df85ee7a57664f14b5ad6e1b2bd8ded4 P 66d7beaf7d8a415f8e5e1b457f1ec504: Log is configured to *not* fsync() on all Append() calls
I20260620 04:25:31.445329 7779 tablet_bootstrap.cc:654] T df85ee7a57664f14b5ad6e1b2bd8ded4 P bde7e8cb28e54e59b808c1bb8439dbf1: Neither blocks nor log segments found. Creating new log.
I20260620 04:25:31.446034 7779 log.cc:824] T df85ee7a57664f14b5ad6e1b2bd8ded4 P bde7e8cb28e54e59b808c1bb8439dbf1: Log is configured to *not* fsync() on all Append() calls
I20260620 04:25:31.446965 7778 tablet_bootstrap.cc:492] T df85ee7a57664f14b5ad6e1b2bd8ded4 P 66d7beaf7d8a415f8e5e1b457f1ec504: No bootstrap required, opened a new log
I20260620 04:25:31.447098 7778 ts_tablet_manager.cc:1397] T df85ee7a57664f14b5ad6e1b2bd8ded4 P 66d7beaf7d8a415f8e5e1b457f1ec504: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 04:25:31.450367 7778 raft_consensus.cc:348] T df85ee7a57664f14b5ad6e1b2bd8ded4 P 66d7beaf7d8a415f8e5e1b457f1ec504 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66d7beaf7d8a415f8e5e1b457f1ec504" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33941 } } peers { permanent_uuid: "449f41f8c68044e29bb5d65d0f0652ef" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42601 } } peers { permanent_uuid: "bde7e8cb28e54e59b808c1bb8439dbf1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38783 } }
I20260620 04:25:31.450580 7778 raft_consensus.cc:374] T df85ee7a57664f14b5ad6e1b2bd8ded4 P 66d7beaf7d8a415f8e5e1b457f1ec504 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:25:31.450613 7778 raft_consensus.cc:729] T df85ee7a57664f14b5ad6e1b2bd8ded4 P 66d7beaf7d8a415f8e5e1b457f1ec504 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66d7beaf7d8a415f8e5e1b457f1ec504, State: Initialized, Role: FOLLOWER
I20260620 04:25:31.451090 7778 consensus_queue.cc:260] T df85ee7a57664f14b5ad6e1b2bd8ded4 P 66d7beaf7d8a415f8e5e1b457f1ec504 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66d7beaf7d8a415f8e5e1b457f1ec504" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33941 } } peers { permanent_uuid: "449f41f8c68044e29bb5d65d0f0652ef" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42601 } } peers { permanent_uuid: "bde7e8cb28e54e59b808c1bb8439dbf1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38783 } }
I20260620 04:25:31.452095 7778 ts_tablet_manager.cc:1428] T df85ee7a57664f14b5ad6e1b2bd8ded4 P 66d7beaf7d8a415f8e5e1b457f1ec504: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 04:25:31.452366 7779 tablet_bootstrap.cc:492] T df85ee7a57664f14b5ad6e1b2bd8ded4 P bde7e8cb28e54e59b808c1bb8439dbf1: No bootstrap required, opened a new log
I20260620 04:25:31.452392 7777 tablet_bootstrap.cc:492] T df85ee7a57664f14b5ad6e1b2bd8ded4 P 449f41f8c68044e29bb5d65d0f0652ef: No bootstrap required, opened a new log
I20260620 04:25:31.452445 7745 heartbeater.cc:503] Master 127.0.98.62:37461 was elected leader, sending a full tablet report...
I20260620 04:25:31.452557 7779 ts_tablet_manager.cc:1397] T df85ee7a57664f14b5ad6e1b2bd8ded4 P bde7e8cb28e54e59b808c1bb8439dbf1: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20260620 04:25:31.452700 7777 ts_tablet_manager.cc:1397] T df85ee7a57664f14b5ad6e1b2bd8ded4 P 449f41f8c68044e29bb5d65d0f0652ef: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.000s
I20260620 04:25:31.456913 7779 raft_consensus.cc:348] T df85ee7a57664f14b5ad6e1b2bd8ded4 P bde7e8cb28e54e59b808c1bb8439dbf1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66d7beaf7d8a415f8e5e1b457f1ec504" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33941 } } peers { permanent_uuid: "449f41f8c68044e29bb5d65d0f0652ef" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42601 } } peers { permanent_uuid: "bde7e8cb28e54e59b808c1bb8439dbf1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38783 } }
I20260620 04:25:31.457229 7779 raft_consensus.cc:374] T df85ee7a57664f14b5ad6e1b2bd8ded4 P bde7e8cb28e54e59b808c1bb8439dbf1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:25:31.457298 7779 raft_consensus.cc:729] T df85ee7a57664f14b5ad6e1b2bd8ded4 P bde7e8cb28e54e59b808c1bb8439dbf1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bde7e8cb28e54e59b808c1bb8439dbf1, State: Initialized, Role: FOLLOWER
I20260620 04:25:31.457715 7779 consensus_queue.cc:260] T df85ee7a57664f14b5ad6e1b2bd8ded4 P bde7e8cb28e54e59b808c1bb8439dbf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66d7beaf7d8a415f8e5e1b457f1ec504" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33941 } } peers { permanent_uuid: "449f41f8c68044e29bb5d65d0f0652ef" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42601 } } peers { permanent_uuid: "bde7e8cb28e54e59b808c1bb8439dbf1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38783 } }
I20260620 04:25:31.459224 7608 heartbeater.cc:503] Master 127.0.98.62:37461 was elected leader, sending a full tablet report...
I20260620 04:25:31.459673 7779 ts_tablet_manager.cc:1428] T df85ee7a57664f14b5ad6e1b2bd8ded4 P bde7e8cb28e54e59b808c1bb8439dbf1: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260620 04:25:31.462287 7777 raft_consensus.cc:348] T df85ee7a57664f14b5ad6e1b2bd8ded4 P 449f41f8c68044e29bb5d65d0f0652ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66d7beaf7d8a415f8e5e1b457f1ec504" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33941 } } peers { permanent_uuid: "449f41f8c68044e29bb5d65d0f0652ef" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42601 } } peers { permanent_uuid: "bde7e8cb28e54e59b808c1bb8439dbf1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38783 } }
I20260620 04:25:31.462860 7777 raft_consensus.cc:374] T df85ee7a57664f14b5ad6e1b2bd8ded4 P 449f41f8c68044e29bb5d65d0f0652ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:25:31.462913 7777 raft_consensus.cc:729] T df85ee7a57664f14b5ad6e1b2bd8ded4 P 449f41f8c68044e29bb5d65d0f0652ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 449f41f8c68044e29bb5d65d0f0652ef, State: Initialized, Role: FOLLOWER
I20260620 04:25:31.463799 7777 consensus_queue.cc:260] T df85ee7a57664f14b5ad6e1b2bd8ded4 P 449f41f8c68044e29bb5d65d0f0652ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66d7beaf7d8a415f8e5e1b457f1ec504" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33941 } } peers { permanent_uuid: "449f41f8c68044e29bb5d65d0f0652ef" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42601 } } peers { permanent_uuid: "bde7e8cb28e54e59b808c1bb8439dbf1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38783 } }
I20260620 04:25:31.464473 7471 heartbeater.cc:503] Master 127.0.98.62:37461 was elected leader, sending a full tablet report...
I20260620 04:25:31.464823 7777 ts_tablet_manager.cc:1428] T df85ee7a57664f14b5ad6e1b2bd8ded4 P 449f41f8c68044e29bb5d65d0f0652ef: Time spent starting tablet: real 0.012s user 0.006s sys 0.000s
I20260620 04:25:31.465138 7783 raft_consensus.cc:482] T df85ee7a57664f14b5ad6e1b2bd8ded4 P 66d7beaf7d8a415f8e5e1b457f1ec504 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:25:31.465276 7783 raft_consensus.cc:504] T df85ee7a57664f14b5ad6e1b2bd8ded4 P 66d7beaf7d8a415f8e5e1b457f1ec504 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66d7beaf7d8a415f8e5e1b457f1ec504" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33941 } } peers { permanent_uuid: "449f41f8c68044e29bb5d65d0f0652ef" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42601 } } peers { permanent_uuid: "bde7e8cb28e54e59b808c1bb8439dbf1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38783 } }
I20260620 04:25:31.466450 7783 leader_election.cc:288] T df85ee7a57664f14b5ad6e1b2bd8ded4 P 66d7beaf7d8a415f8e5e1b457f1ec504 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 449f41f8c68044e29bb5d65d0f0652ef (127.0.98.1:42601), bde7e8cb28e54e59b808c1bb8439dbf1 (127.0.98.2:38783)
I20260620 04:25:31.471107 7775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:31.467182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55935 (local address 127.0.98.2:38783)
0620 04:25:31.467293 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:31.467297 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:31.467342 (+ 45us) server_negotiation.cc:407] Connection header received
0620 04:25:31.467469 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:31.467499 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:31.467548 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:31.467610 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:25:31.467988 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:31.468684 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:31.469988 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:31.470542 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:31.470636 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:31.470686 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:31.470913 (+ 227us) server_negotiation.cc:299] Negotiation successful
0620 04:25:31.470982 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 04:25:31.471136 7752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:31.467127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38783 (local address 127.0.98.3:55935)
0620 04:25:31.467280 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:31.467293 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:31.467374 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:31.467719 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:31.467722 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:31.467738 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:25:31.467839 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:31.467845 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:31.468797 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:31.468800 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:31.469890 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:31.469898 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:31.470639 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:31.470643 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:31.470777 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:31.470817 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 04:25:31.470897 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 04:25:31.470971 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"mutex_wait_us":23}
I20260620 04:25:31.471161 7776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:31.466810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53927 (local address 127.0.98.1:42601)
0620 04:25:31.466929 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:31.466933 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:31.467133 (+ 200us) server_negotiation.cc:407] Connection header received
0620 04:25:31.467175 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:31.467178 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:31.467232 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:31.467293 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:25:31.467659 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:31.468387 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:31.469907 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:31.470495 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:31.470598 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:31.470659 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:31.470931 (+ 272us) server_negotiation.cc:299] Negotiation successful
0620 04:25:31.470990 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46,"spinlock_wait_cycles":3072}
I20260620 04:25:31.471179 7758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:31.466613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42601 (local address 127.0.98.3:53927)
0620 04:25:31.466910 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:31.466925 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:31.467020 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:31.467411 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:31.467414 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:31.467431 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:25:31.467545 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:31.467552 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:31.468491 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:31.468494 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:31.469796 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:31.469806 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:31.470617 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:31.470621 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:31.470774 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:31.470823 (+ 49us) client_negotiation.cc:769] Sending connection context
0620 04:25:31.470918 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 04:25:31.470970 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"mutex_wait_us":37}
I20260620 04:25:31.471864 7424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df85ee7a57664f14b5ad6e1b2bd8ded4" candidate_uuid: "66d7beaf7d8a415f8e5e1b457f1ec504" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "449f41f8c68044e29bb5d65d0f0652ef" is_pre_election: true
I20260620 04:25:31.471947 7561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df85ee7a57664f14b5ad6e1b2bd8ded4" candidate_uuid: "66d7beaf7d8a415f8e5e1b457f1ec504" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bde7e8cb28e54e59b808c1bb8439dbf1" is_pre_election: true
I20260620 04:25:31.472262 7424 raft_consensus.cc:2454] T df85ee7a57664f14b5ad6e1b2bd8ded4 P 449f41f8c68044e29bb5d65d0f0652ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66d7beaf7d8a415f8e5e1b457f1ec504 in term 0.
I20260620 04:25:31.472262 7561 raft_consensus.cc:2454] T df85ee7a57664f14b5ad6e1b2bd8ded4 P bde7e8cb28e54e59b808c1bb8439dbf1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66d7beaf7d8a415f8e5e1b457f1ec504 in term 0.
I20260620 04:25:31.472666 7633 leader_election.cc:302] T df85ee7a57664f14b5ad6e1b2bd8ded4 P 66d7beaf7d8a415f8e5e1b457f1ec504 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66d7beaf7d8a415f8e5e1b457f1ec504, bde7e8cb28e54e59b808c1bb8439dbf1; no voters:
I20260620 04:25:31.472893 7783 raft_consensus.cc:2781] T df85ee7a57664f14b5ad6e1b2bd8ded4 P 66d7beaf7d8a415f8e5e1b457f1ec504 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 04:25:31.472981 7783 raft_consensus.cc:482] T df85ee7a57664f14b5ad6e1b2bd8ded4 P 66d7beaf7d8a415f8e5e1b457f1ec504 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 04:25:31.473033 7783 raft_consensus.cc:3037] T df85ee7a57664f14b5ad6e1b2bd8ded4 P 66d7beaf7d8a415f8e5e1b457f1ec504 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:25:31.474027 7783 raft_consensus.cc:504] T df85ee7a57664f14b5ad6e1b2bd8ded4 P 66d7beaf7d8a415f8e5e1b457f1ec504 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66d7beaf7d8a415f8e5e1b457f1ec504" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33941 } } peers { permanent_uuid: "449f41f8c68044e29bb5d65d0f0652ef" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42601 } } peers { permanent_uuid: "bde7e8cb28e54e59b808c1bb8439dbf1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38783 } }
I20260620 04:25:31.474363 7783 leader_election.cc:288] T df85ee7a57664f14b5ad6e1b2bd8ded4 P 66d7beaf7d8a415f8e5e1b457f1ec504 [CANDIDATE]: Term 1 election: Requested vote from peers 449f41f8c68044e29bb5d65d0f0652ef (127.0.98.1:42601), bde7e8cb28e54e59b808c1bb8439dbf1 (127.0.98.2:38783)
I20260620 04:25:31.474654 7424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df85ee7a57664f14b5ad6e1b2bd8ded4" candidate_uuid: "66d7beaf7d8a415f8e5e1b457f1ec504" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "449f41f8c68044e29bb5d65d0f0652ef"
I20260620 04:25:31.474766 7424 raft_consensus.cc:3037] T df85ee7a57664f14b5ad6e1b2bd8ded4 P 449f41f8c68044e29bb5d65d0f0652ef [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:25:31.474797 7561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df85ee7a57664f14b5ad6e1b2bd8ded4" candidate_uuid: "66d7beaf7d8a415f8e5e1b457f1ec504" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bde7e8cb28e54e59b808c1bb8439dbf1"
I20260620 04:25:31.474890 7561 raft_consensus.cc:3037] T df85ee7a57664f14b5ad6e1b2bd8ded4 P bde7e8cb28e54e59b808c1bb8439dbf1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:25:31.476013 7424 raft_consensus.cc:2454] T df85ee7a57664f14b5ad6e1b2bd8ded4 P 449f41f8c68044e29bb5d65d0f0652ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66d7beaf7d8a415f8e5e1b457f1ec504 in term 1.
I20260620 04:25:31.476013 7561 raft_consensus.cc:2454] T df85ee7a57664f14b5ad6e1b2bd8ded4 P bde7e8cb28e54e59b808c1bb8439dbf1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66d7beaf7d8a415f8e5e1b457f1ec504 in term 1.
I20260620 04:25:31.476265 7633 leader_election.cc:302] T df85ee7a57664f14b5ad6e1b2bd8ded4 P 66d7beaf7d8a415f8e5e1b457f1ec504 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66d7beaf7d8a415f8e5e1b457f1ec504, bde7e8cb28e54e59b808c1bb8439dbf1; no voters:
I20260620 04:25:31.476415 7783 raft_consensus.cc:2781] T df85ee7a57664f14b5ad6e1b2bd8ded4 P 66d7beaf7d8a415f8e5e1b457f1ec504 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:25:31.476635 7783 raft_consensus.cc:686] T df85ee7a57664f14b5ad6e1b2bd8ded4 P 66d7beaf7d8a415f8e5e1b457f1ec504 [term 1 LEADER]: Becoming Leader. State: Replica: 66d7beaf7d8a415f8e5e1b457f1ec504, State: Running, Role: LEADER
I20260620 04:25:31.476975 7783 consensus_queue.cc:237] T df85ee7a57664f14b5ad6e1b2bd8ded4 P 66d7beaf7d8a415f8e5e1b457f1ec504 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66d7beaf7d8a415f8e5e1b457f1ec504" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33941 } } peers { permanent_uuid: "449f41f8c68044e29bb5d65d0f0652ef" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42601 } } peers { permanent_uuid: "bde7e8cb28e54e59b808c1bb8439dbf1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38783 } }
I20260620 04:25:31.480216 6508 catalog_manager.cc:5495] T df85ee7a57664f14b5ad6e1b2bd8ded4 P 66d7beaf7d8a415f8e5e1b457f1ec504 reported cstate change: term changed from 0 to 1, leader changed from <none> to 66d7beaf7d8a415f8e5e1b457f1ec504 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "66d7beaf7d8a415f8e5e1b457f1ec504" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66d7beaf7d8a415f8e5e1b457f1ec504" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "449f41f8c68044e29bb5d65d0f0652ef" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bde7e8cb28e54e59b808c1bb8439dbf1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38783 } health_report { overall_health: UNKNOWN } } }
W20260620 04:25:31.483183 7473 tablet.cc:2369] T df85ee7a57664f14b5ad6e1b2bd8ded4 P 449f41f8c68044e29bb5d65d0f0652ef: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 04:25:31.492800 7774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:31.489571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60592 (local address 127.0.98.3:33941)
0620 04:25:31.489687 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:31.489691 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:31.489705 (+ 14us) server_negotiation.cc:407] Connection header received
0620 04:25:31.489806 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:31.489808 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:31.489861 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:31.489951 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:25:31.490184 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:31.490699 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:31.491510 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:31.491713 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:31.491795 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:31.492112 (+ 317us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:25:31.492507 (+ 395us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:25:31.492604 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:31.492650 (+ 46us) server_negotiation.cc:299] Negotiation successful
0620 04:25:31.492699 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 04:25:31.498184 7561 raft_consensus.cc:1261] T df85ee7a57664f14b5ad6e1b2bd8ded4 P bde7e8cb28e54e59b808c1bb8439dbf1 [term 1 FOLLOWER]: Refusing update from remote peer 66d7beaf7d8a415f8e5e1b457f1ec504: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:25:31.498184 7424 raft_consensus.cc:1261] T df85ee7a57664f14b5ad6e1b2bd8ded4 P 449f41f8c68044e29bb5d65d0f0652ef [term 1 FOLLOWER]: Refusing update from remote peer 66d7beaf7d8a415f8e5e1b457f1ec504: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:25:31.498859 7786 consensus_queue.cc:1035] T df85ee7a57664f14b5ad6e1b2bd8ded4 P 66d7beaf7d8a415f8e5e1b457f1ec504 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bde7e8cb28e54e59b808c1bb8439dbf1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:25:31.499059 7783 consensus_queue.cc:1035] T df85ee7a57664f14b5ad6e1b2bd8ded4 P 66d7beaf7d8a415f8e5e1b457f1ec504 [LEADER]: Connected to new peer: Peer: permanent_uuid: "449f41f8c68044e29bb5d65d0f0652ef" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:25:31.505373 7789 mvcc.cc:205] Tried to move back new op lower bound from 7298783361009893376 to 7298783360931946496. Current Snapshot: MvccSnapshot[applied={T|T < 7298783361009893376}]
I20260620 04:25:31.508553 7790 mvcc.cc:205] Tried to move back new op lower bound from 7298783361009893376 to 7298783360931946496. Current Snapshot: MvccSnapshot[applied={T|T < 7298783361009893376 or (T in {7298783361009893376})}]
I20260620 04:25:31.514612 6507 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44270:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:25:31.514768 6507 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44270:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:25:31.518832 6507 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 961f73cd6ac541f79ace3f8d78ff255b: Sending DeleteTablet for 3 replicas of tablet df85ee7a57664f14b5ad6e1b2bd8ded4
I20260620 04:25:31.520025 7541 tablet_service.cc:1514] Processing DeleteTablet for tablet df85ee7a57664f14b5ad6e1b2bd8ded4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:25:31 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36522
I20260620 04:25:31.520530 7404 tablet_service.cc:1514] Processing DeleteTablet for tablet df85ee7a57664f14b5ad6e1b2bd8ded4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:25:31 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59298
I20260620 04:25:31.521023 7678 tablet_service.cc:1514] Processing DeleteTablet for tablet df85ee7a57664f14b5ad6e1b2bd8ded4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:25:31 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60584
I20260620 04:25:31.521133 7801 tablet_replica.cc:330] T df85ee7a57664f14b5ad6e1b2bd8ded4 P 449f41f8c68044e29bb5d65d0f0652ef: stopping tablet replica
I20260620 04:25:31.521454 7801 raft_consensus.cc:2229] T df85ee7a57664f14b5ad6e1b2bd8ded4 P 449f41f8c68044e29bb5d65d0f0652ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:25:31.521458 7802 tablet_replica.cc:330] T df85ee7a57664f14b5ad6e1b2bd8ded4 P 66d7beaf7d8a415f8e5e1b457f1ec504: stopping tablet replica
I20260620 04:25:31.521785 7801 raft_consensus.cc:2258] T df85ee7a57664f14b5ad6e1b2bd8ded4 P 449f41f8c68044e29bb5d65d0f0652ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:25:31.521807 7802 raft_consensus.cc:2229] T df85ee7a57664f14b5ad6e1b2bd8ded4 P 66d7beaf7d8a415f8e5e1b457f1ec504 [term 1 LEADER]: Raft consensus shutting down.
I20260620 04:25:31.522219 7802 raft_consensus.cc:2258] T df85ee7a57664f14b5ad6e1b2bd8ded4 P 66d7beaf7d8a415f8e5e1b457f1ec504 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:25:31.523010 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 6476
I20260620 04:25:31.524132 7801 ts_tablet_manager.cc:1905] T df85ee7a57664f14b5ad6e1b2bd8ded4 P 449f41f8c68044e29bb5d65d0f0652ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:25:31.524367 7802 ts_tablet_manager.cc:1905] T df85ee7a57664f14b5ad6e1b2bd8ded4 P 66d7beaf7d8a415f8e5e1b457f1ec504: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:25:31.526782 7801 ts_tablet_manager.cc:1918] T df85ee7a57664f14b5ad6e1b2bd8ded4 P 449f41f8c68044e29bb5d65d0f0652ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:25:31.526876 7801 log.cc:1196] T df85ee7a57664f14b5ad6e1b2bd8ded4 P 449f41f8c68044e29bb5d65d0f0652ef: Deleting WAL directory at /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/ts-0/wal/wals/df85ee7a57664f14b5ad6e1b2bd8ded4
I20260620 04:25:31.527102 7802 ts_tablet_manager.cc:1918] T df85ee7a57664f14b5ad6e1b2bd8ded4 P 66d7beaf7d8a415f8e5e1b457f1ec504: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:25:31.527197 7801 ts_tablet_manager.cc:1939] T df85ee7a57664f14b5ad6e1b2bd8ded4 P 449f41f8c68044e29bb5d65d0f0652ef: Deleting consensus metadata
I20260620 04:25:31.527302 7802 log.cc:1196] T df85ee7a57664f14b5ad6e1b2bd8ded4 P 66d7beaf7d8a415f8e5e1b457f1ec504: Deleting WAL directory at /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/ts-2/wal/wals/df85ee7a57664f14b5ad6e1b2bd8ded4
I20260620 04:25:31.527633 7802 ts_tablet_manager.cc:1939] T df85ee7a57664f14b5ad6e1b2bd8ded4 P 66d7beaf7d8a415f8e5e1b457f1ec504: Deleting consensus metadata
W20260620 04:25:31.529675 7358 connection.cc:535] server connection from 127.0.0.1:59298 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260620 04:25:31.530179 7634 connection.cc:535] server connection from 127.0.0.1:60584 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 04:25:31.531239 7799 tablet_replica.cc:330] T df85ee7a57664f14b5ad6e1b2bd8ded4 P bde7e8cb28e54e59b808c1bb8439dbf1: stopping tablet replica
I20260620 04:25:31.531625 7799 raft_consensus.cc:2229] T df85ee7a57664f14b5ad6e1b2bd8ded4 P bde7e8cb28e54e59b808c1bb8439dbf1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:25:31.531985 7799 raft_consensus.cc:2258] T df85ee7a57664f14b5ad6e1b2bd8ded4 P bde7e8cb28e54e59b808c1bb8439dbf1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:25:31.533998 7799 ts_tablet_manager.cc:1905] T df85ee7a57664f14b5ad6e1b2bd8ded4 P bde7e8cb28e54e59b808c1bb8439dbf1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:25:31.536278 7799 ts_tablet_manager.cc:1918] T df85ee7a57664f14b5ad6e1b2bd8ded4 P bde7e8cb28e54e59b808c1bb8439dbf1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:25:31.536370 7799 log.cc:1196] T df85ee7a57664f14b5ad6e1b2bd8ded4 P bde7e8cb28e54e59b808c1bb8439dbf1: Deleting WAL directory at /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/ts-1/wal/wals/df85ee7a57664f14b5ad6e1b2bd8ded4
I20260620 04:25:31.536650 7799 ts_tablet_manager.cc:1939] T df85ee7a57664f14b5ad6e1b2bd8ded4 P bde7e8cb28e54e59b808c1bb8439dbf1: Deleting consensus metadata
W20260620 04:25:31.537904 7494 connection.cc:441] server connection from 127.0.0.1:36522 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36522 (request call id 1) could send its response
I20260620 04:25:31.540953 7230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:31.520854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49078 (local address 127.0.98.61:34531)
0620 04:25:31.520959 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:31.520963 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:31.530964 (+ 10001us) server_negotiation.cc:407] Connection header received
0620 04:25:31.531090 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:31.531093 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:31.531142 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:31.531205 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:31.532331 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:31.533140 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:31.534359 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:31.534698 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:31.534762 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:31.536878 (+ 2116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:31.536882 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:31.536885 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:25:31.536913 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:31.538508 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:31.539610 (+ 1102us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:31.539614 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:31.539617 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:31.539704 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:31.540025 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:31.540029 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:31.540031 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:31.540242 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:31.540361 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:31.540693 (+ 332us) server_negotiation.cc:299] Negotiation successful
0620 04:25:31.540803 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 04:25:31.546972 7614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:31.538723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44158 (local address 127.0.98.60:45713)
0620 04:25:31.538861 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:31.538865 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:31.538989 (+ 124us) server_negotiation.cc:407] Connection header received
0620 04:25:31.539100 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:31.539103 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:31.539150 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:31.539229 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:31.539864 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:31.540613 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:31.541633 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:31.541953 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:31.542017 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:31.543687 (+ 1670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:31.543691 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:31.543694 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:25:31.543718 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:31.545174 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:31.545644 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:31.545648 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:31.545649 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:31.545738 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:31.546033 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:31.546036 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:31.546038 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:31.546244 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:31.546352 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:31.546696 (+ 344us) server_negotiation.cc:299] Negotiation successful
0620 04:25:31.546817 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 04:25:31.551414 7477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:31.521435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44144 (local address 127.0.98.60:45713)
0620 04:25:31.521547 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:31.521552 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:31.542620 (+ 21068us) server_negotiation.cc:407] Connection header received
0620 04:25:31.542807 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:31.542810 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:31.542854 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:31.542923 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:31.543725 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:31.544658 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:31.545722 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:31.546006 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:31.546074 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:31.547874 (+ 1800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:31.547878 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:31.547881 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:25:31.547904 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:31.549641 (+ 1737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:31.550079 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:31.550083 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:31.550084 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:31.550161 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:31.550447 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:31.550451 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:31.550453 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:31.550685 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:31.550811 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:31.551147 (+ 336us) server_negotiation.cc:299] Negotiation successful
0620 04:25:31.551261 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 04:25:31.552728 7757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:31.533026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49090 (local address 127.0.98.61:34531)
0620 04:25:31.533138 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:31.533142 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:31.538487 (+ 5345us) server_negotiation.cc:407] Connection header received
0620 04:25:31.538652 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:31.538655 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:31.538706 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:31.538753 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:31.539551 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:31.540337 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:31.541482 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:31.541802 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:31.541866 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:31.543687 (+ 1821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:31.543691 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:31.543694 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:25:31.543718 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:31.550084 (+ 6366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:31.551505 (+ 1421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:31.551508 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:31.551510 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:31.551597 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:31.551856 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:31.551860 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:31.551862 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:31.552118 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:31.552229 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:31.552452 (+ 223us) server_negotiation.cc:299] Negotiation successful
0620 04:25:31.552574 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 04:25:32.509559 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:32.509085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37461 (local address 127.0.98.1:41869)
0620 04:25:32.509347 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:32.509448 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":79,"threads_started":1}
W20260620 04:25:32.509999 7471 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37461 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:37461: connect: Connection refused (error 111)
I20260620 04:25:32.514183 7811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:32.513734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37461 (local address 127.0.98.2:50569)
0620 04:25:32.513970 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:32.514062 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":79,"threads_started":1}
W20260620 04:25:32.514487 7608 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37461 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:37461: connect: Connection refused (error 111)
I20260620 04:25:32.520089 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:32.519662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37461 (local address 127.0.98.3:45863)
0620 04:25:32.519883 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:32.519975 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":62,"threads_started":1}
W20260620 04:25:32.520347 7745 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37461 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:37461: connect: Connection refused (error 111)
I20260620 04:25:33.018980 7813 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b026c99d08842f5a61f0a658e08398a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 961f73cd6ac541f79ace3f8d78ff255b)
I20260620 04:25:33.019177 7813 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b026c99d08842f5a61f0a658e08398a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "961f73cd6ac541f79ace3f8d78ff255b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37461 } } peers { permanent_uuid: "1adca5af9a0644fcb8f274cbc0ae7292" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34531 } attrs { promote: false } } peers { permanent_uuid: "3b026c99d08842f5a61f0a658e08398a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45713 } attrs { promote: false } }
I20260620 04:25:33.020058 7814 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1adca5af9a0644fcb8f274cbc0ae7292 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 961f73cd6ac541f79ace3f8d78ff255b)
I20260620 04:25:33.020242 7814 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1adca5af9a0644fcb8f274cbc0ae7292 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "961f73cd6ac541f79ace3f8d78ff255b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37461 } } peers { permanent_uuid: "1adca5af9a0644fcb8f274cbc0ae7292" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34531 } attrs { promote: false } } peers { permanent_uuid: "3b026c99d08842f5a61f0a658e08398a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45713 } attrs { promote: false } }
I20260620 04:25:33.020372 7813 leader_election.cc:288] T 00000000000000000000000000000000 P 3b026c99d08842f5a61f0a658e08398a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 961f73cd6ac541f79ace3f8d78ff255b (127.0.98.62:37461), 1adca5af9a0644fcb8f274cbc0ae7292 (127.0.98.61:34531)
I20260620 04:25:33.021088 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:33.020699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37461 (local address 127.0.0.1:44738)
0620 04:25:33.020908 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:33.020992 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":63,"threads_started":1}
W20260620 04:25:33.021445 7278 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37461: connect: Connection refused (error 111)
I20260620 04:25:33.021754 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:33.021362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37461 (local address 127.0.0.1:44746)
0620 04:25:33.021594 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:33.021664 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":56,"threads_started":1}
W20260620 04:25:33.021989 7095 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37461: connect: Connection refused (error 111)
I20260620 04:25:33.022482 7814 leader_election.cc:288] T 00000000000000000000000000000000 P 1adca5af9a0644fcb8f274cbc0ae7292 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 961f73cd6ac541f79ace3f8d78ff255b (127.0.98.62:37461), 3b026c99d08842f5a61f0a658e08398a (127.0.98.60:45713)
I20260620 04:25:33.022728 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:33.022524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37461 (local address 127.0.0.1:44750)
0620 04:25:33.022613 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:33.022663 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260620 04:25:33.022965 7095 leader_election.cc:334] T 00000000000000000000000000000000 P 1adca5af9a0644fcb8f274cbc0ae7292 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 961f73cd6ac541f79ace3f8d78ff255b (127.0.98.62:37461): Network error: Client connection negotiation failed: client connection to 127.0.98.62:37461: connect: Connection refused (error 111)
I20260620 04:25:33.023042 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:33.022852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37461 (local address 127.0.0.1:44754)
0620 04:25:33.022933 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:33.022978 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260620 04:25:33.023144 7278 leader_election.cc:334] T 00000000000000000000000000000000 P 3b026c99d08842f5a61f0a658e08398a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 961f73cd6ac541f79ace3f8d78ff255b (127.0.98.62:37461): Network error: Client connection negotiation failed: client connection to 127.0.98.62:37461: connect: Connection refused (error 111)
I20260620 04:25:33.024495 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:33.020451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34531 (local address 127.0.0.1:49096)
0620 04:25:33.020840 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:33.020856 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:33.020977 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:33.021259 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:33.021262 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:33.021279 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:25:33.021389 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:33.021394 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:33.022358 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:33.022361 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:33.023514 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:33.023521 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:33.024185 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:33.024188 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:33.024277 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:33.024300 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 04:25:33.024354 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 04:25:33.024409 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":69,"threads_started":1}
I20260620 04:25:33.024559 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:33.020511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49096 (local address 127.0.98.61:34531)
0620 04:25:33.020789 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:33.020794 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:33.020948 (+ 154us) server_negotiation.cc:407] Connection header received
0620 04:25:33.021105 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:33.021109 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:33.021163 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:33.021249 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:25:33.021522 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:33.022240 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:33.023621 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:33.024077 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:33.024153 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:33.024201 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:33.024378 (+ 177us) server_negotiation.cc:299] Negotiation successful
0620 04:25:33.024449 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":94,"threads_started":1}
I20260620 04:25:33.024962 7115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3b026c99d08842f5a61f0a658e08398a" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "1adca5af9a0644fcb8f274cbc0ae7292" is_pre_election: true
I20260620 04:25:33.025156 7115 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1adca5af9a0644fcb8f274cbc0ae7292 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b026c99d08842f5a61f0a658e08398a in term 1.
I20260620 04:25:33.025449 7276 leader_election.cc:302] T 00000000000000000000000000000000 P 3b026c99d08842f5a61f0a658e08398a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1adca5af9a0644fcb8f274cbc0ae7292, 3b026c99d08842f5a61f0a658e08398a; no voters: 961f73cd6ac541f79ace3f8d78ff255b
I20260620 04:25:33.025643 7813 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b026c99d08842f5a61f0a658e08398a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 04:25:33.025698 7813 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b026c99d08842f5a61f0a658e08398a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 961f73cd6ac541f79ace3f8d78ff255b)
I20260620 04:25:33.025741 7813 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b026c99d08842f5a61f0a658e08398a [term 1 FOLLOWER]: Advancing to term 2
I20260620 04:25:33.026297 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:33.022322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44160 (local address 127.0.98.60:45713)
0620 04:25:33.022758 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:33.022839 (+ 81us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:33.022854 (+ 15us) server_negotiation.cc:407] Connection header received
0620 04:25:33.022893 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:33.022895 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:33.022935 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:33.023240 (+ 305us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:25:33.023612 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:33.024149 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:33.025306 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:33.025840 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:33.025929 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:33.025980 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:33.026133 (+ 153us) server_negotiation.cc:299] Negotiation successful
0620 04:25:33.026197 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":95,"threads_started":1}
I20260620 04:25:33.026296 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:33.021768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45713 (local address 127.0.0.1:44160)
0620 04:25:33.021878 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:33.021890 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:33.021969 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:33.023223 (+ 1254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:33.023226 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:33.023431 (+ 205us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:25:33.023505 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:33.023511 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:33.024240 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:33.024243 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:33.025182 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:33.025191 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:33.025937 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:33.025940 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:33.026028 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:33.026049 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 04:25:33.026112 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 04:25:33.026178 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63,"thread_start_us":56,"threads_started":1}
I20260620 04:25:33.026726 7301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1adca5af9a0644fcb8f274cbc0ae7292" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "3b026c99d08842f5a61f0a658e08398a" is_pre_election: true
I20260620 04:25:33.026886 7813 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b026c99d08842f5a61f0a658e08398a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "961f73cd6ac541f79ace3f8d78ff255b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37461 } } peers { permanent_uuid: "1adca5af9a0644fcb8f274cbc0ae7292" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34531 } attrs { promote: false } } peers { permanent_uuid: "3b026c99d08842f5a61f0a658e08398a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45713 } attrs { promote: false } }
I20260620 04:25:33.027117 7301 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 3b026c99d08842f5a61f0a658e08398a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1adca5af9a0644fcb8f274cbc0ae7292 in current term 2: Already voted for candidate 3b026c99d08842f5a61f0a658e08398a in this term.
I20260620 04:25:33.027261 7813 leader_election.cc:288] T 00000000000000000000000000000000 P 3b026c99d08842f5a61f0a658e08398a [CANDIDATE]: Term 2 election: Requested vote from peers 961f73cd6ac541f79ace3f8d78ff255b (127.0.98.62:37461), 1adca5af9a0644fcb8f274cbc0ae7292 (127.0.98.61:34531)
I20260620 04:25:33.027576 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:33.027348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37461 (local address 127.0.0.1:44770)
0620 04:25:33.027473 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:33.027525 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260620 04:25:33.027570 7115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3b026c99d08842f5a61f0a658e08398a" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "1adca5af9a0644fcb8f274cbc0ae7292"
I20260620 04:25:33.027652 7095 leader_election.cc:302] T 00000000000000000000000000000000 P 1adca5af9a0644fcb8f274cbc0ae7292 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1adca5af9a0644fcb8f274cbc0ae7292; no voters: 3b026c99d08842f5a61f0a658e08398a, 961f73cd6ac541f79ace3f8d78ff255b
I20260620 04:25:33.027704 7115 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1adca5af9a0644fcb8f274cbc0ae7292 [term 1 FOLLOWER]: Advancing to term 2
I20260620 04:25:33.028213 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:33.028001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37461 (local address 127.0.0.1:44774)
0620 04:25:33.028105 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:33.028153 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260620 04:25:33.028308 7278 leader_election.cc:334] T 00000000000000000000000000000000 P 3b026c99d08842f5a61f0a658e08398a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 961f73cd6ac541f79ace3f8d78ff255b (127.0.98.62:37461): Network error: Client connection negotiation failed: client connection to 127.0.98.62:37461: connect: Connection refused (error 111)
I20260620 04:25:33.028786 7115 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1adca5af9a0644fcb8f274cbc0ae7292 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b026c99d08842f5a61f0a658e08398a in term 2.
I20260620 04:25:33.028887 7814 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 1adca5af9a0644fcb8f274cbc0ae7292 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 04:25:33.029124 7276 leader_election.cc:302] T 00000000000000000000000000000000 P 3b026c99d08842f5a61f0a658e08398a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1adca5af9a0644fcb8f274cbc0ae7292, 3b026c99d08842f5a61f0a658e08398a; no voters: 961f73cd6ac541f79ace3f8d78ff255b
I20260620 04:25:33.029270 7813 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b026c99d08842f5a61f0a658e08398a [term 2 FOLLOWER]: Leader election won for term 2
I20260620 04:25:33.029456 7813 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3b026c99d08842f5a61f0a658e08398a [term 2 LEADER]: Becoming Leader. State: Replica: 3b026c99d08842f5a61f0a658e08398a, State: Running, Role: LEADER
I20260620 04:25:33.029726 7813 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b026c99d08842f5a61f0a658e08398a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "961f73cd6ac541f79ace3f8d78ff255b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37461 } } peers { permanent_uuid: "1adca5af9a0644fcb8f274cbc0ae7292" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34531 } attrs { promote: false } } peers { permanent_uuid: "3b026c99d08842f5a61f0a658e08398a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45713 } attrs { promote: false } }
I20260620 04:25:33.031025 7823 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b026c99d08842f5a61f0a658e08398a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b026c99d08842f5a61f0a658e08398a. Latest consensus state: current_term: 2 leader_uuid: "3b026c99d08842f5a61f0a658e08398a" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "961f73cd6ac541f79ace3f8d78ff255b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37461 } } peers { permanent_uuid: "1adca5af9a0644fcb8f274cbc0ae7292" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34531 } attrs { promote: false } } peers { permanent_uuid: "3b026c99d08842f5a61f0a658e08398a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45713 } attrs { promote: false } } }
I20260620 04:25:33.031128 7823 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b026c99d08842f5a61f0a658e08398a [sys.catalog]: This master's current role is: LEADER
I20260620 04:25:33.031381 7825 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:25:33.033437 7825 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:25:33.033797 7825 catalog_manager.cc:1259] Loaded cluster ID: 65e72dfb217a4b24b6c29af381fead6f
I20260620 04:25:33.033839 7825 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:25:33.034364 7825 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:25:33.034734 7825 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3b026c99d08842f5a61f0a658e08398a: Loaded TSK: 0
I20260620 04:25:33.035413 7825 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 04:25:33.056120 7290 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44158:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 04:25:33.059177 7290 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 04:25:33.062109 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:33.061880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37461 (local address 127.0.0.1:44786)
0620 04:25:33.061981 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:33.062036 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260620 04:25:33.062469 7115 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1adca5af9a0644fcb8f274cbc0ae7292 [term 2 FOLLOWER]: Refusing update from remote peer 3b026c99d08842f5a61f0a658e08398a: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260620 04:25:33.062716 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:33.062528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37461 (local address 127.0.0.1:44788)
0620 04:25:33.062620 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:33.062664 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260620 04:25:33.062830 7278 consensus_peers.cc:487] T 00000000000000000000000000000000 P 3b026c99d08842f5a61f0a658e08398a -> Peer 961f73cd6ac541f79ace3f8d78ff255b (127.0.98.62:37461): Couldn't send request to peer 961f73cd6ac541f79ace3f8d78ff255b. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37461: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 04:25:33.063027 7823 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3b026c99d08842f5a61f0a658e08398a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1adca5af9a0644fcb8f274cbc0ae7292" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34531 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260620 04:25:33.064462 7814 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1adca5af9a0644fcb8f274cbc0ae7292 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b026c99d08842f5a61f0a658e08398a. Latest consensus state: current_term: 2 leader_uuid: "3b026c99d08842f5a61f0a658e08398a" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "961f73cd6ac541f79ace3f8d78ff255b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37461 } } peers { permanent_uuid: "1adca5af9a0644fcb8f274cbc0ae7292" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34531 } attrs { promote: false } } peers { permanent_uuid: "3b026c99d08842f5a61f0a658e08398a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45713 } attrs { promote: false } } }
I20260620 04:25:33.064571 7814 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1adca5af9a0644fcb8f274cbc0ae7292 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 04:25:33.065469 7823 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b026c99d08842f5a61f0a658e08398a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "3b026c99d08842f5a61f0a658e08398a" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "961f73cd6ac541f79ace3f8d78ff255b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37461 } } peers { permanent_uuid: "1adca5af9a0644fcb8f274cbc0ae7292" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34531 } attrs { promote: false } } peers { permanent_uuid: "3b026c99d08842f5a61f0a658e08398a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45713 } attrs { promote: false } } }
I20260620 04:25:33.065567 7823 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b026c99d08842f5a61f0a658e08398a [sys.catalog]: This master's current role is: LEADER
I20260620 04:25:33.066251 7814 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1adca5af9a0644fcb8f274cbc0ae7292 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "3b026c99d08842f5a61f0a658e08398a" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "961f73cd6ac541f79ace3f8d78ff255b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37461 } } peers { permanent_uuid: "1adca5af9a0644fcb8f274cbc0ae7292" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34531 } attrs { promote: false } } peers { permanent_uuid: "3b026c99d08842f5a61f0a658e08398a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45713 } attrs { promote: false } } }
I20260620 04:25:33.066358 7814 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1adca5af9a0644fcb8f274cbc0ae7292 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 04:25:33.075307 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:33.071125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42601 (local address 127.0.0.1:59314)
0620 04:25:33.071223 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:33.071237 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:33.071311 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:33.071741 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:33.071745 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:33.071764 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:25:33.071869 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:33.071876 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:33.072885 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:33.072888 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:33.073994 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:33.074004 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:33.074930 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:33.074933 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:33.075017 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:33.075041 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 04:25:33.075113 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 04:25:33.075180 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260620 04:25:33.075347 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:33.071218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59314 (local address 127.0.98.1:42601)
0620 04:25:33.071503 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:33.071507 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:33.071519 (+ 12us) server_negotiation.cc:407] Connection header received
0620 04:25:33.071567 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:33.071570 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:33.071633 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:33.071712 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:25:33.072071 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:33.072797 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:33.074146 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:33.074825 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:33.074940 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:33.075012 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:33.075128 (+ 116us) server_negotiation.cc:299] Negotiation successful
0620 04:25:33.075202 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":75,"threads_started":1}
I20260620 04:25:33.076033 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:33.071362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33941 (local address 127.0.0.1:60604)
0620 04:25:33.071573 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:33.071585 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:33.071653 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:33.072476 (+ 823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:33.072479 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:33.072495 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:25:33.072586 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:33.072592 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:33.073502 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:33.073506 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:33.074627 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:33.074637 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:33.075615 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:33.075619 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:33.075737 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:33.075759 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 04:25:33.075815 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 04:25:33.075882 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"mutex_wait_us":26}
I20260620 04:25:33.076041 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:33.071903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38783 (local address 127.0.0.1:36534)
0620 04:25:33.072146 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:33.072158 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:33.072230 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:33.072605 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:33.072609 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:33.072626 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:25:33.072735 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:33.072741 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:33.073919 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:33.073922 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:33.074713 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:33.074724 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:33.075568 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:33.075572 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:33.075680 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:33.075735 (+ 55us) client_negotiation.cc:769] Sending connection context
0620 04:25:33.075790 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 04:25:33.075866 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"mutex_wait_us":27,"thread_start_us":114,"threads_started":1}
I20260620 04:25:33.076257 7404 tablet_service.cc:1467] Processing CreateTablet for tablet 4ee163af0bf64b29a9bc33df55a960c5 (DEFAULT_TABLE table=test-table [id=c3e9a1127998473cb79f45fe83381b0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:25:33.076429 7835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:33.072021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36534 (local address 127.0.98.2:38783)
0620 04:25:33.072381 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:33.072384 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:33.072394 (+ 10us) server_negotiation.cc:407] Connection header received
0620 04:25:33.072434 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:33.072437 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:33.072478 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:33.072592 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:25:33.073066 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:33.073823 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:33.074819 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:33.075422 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:33.076125 (+ 703us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:33.076192 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:33.076230 (+ 38us) server_negotiation.cc:299] Negotiation successful
0620 04:25:33.076301 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":65,"threads_started":1}
I20260620 04:25:33.076551 7404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ee163af0bf64b29a9bc33df55a960c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:25:33.076781 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:33.071612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60604 (local address 127.0.98.3:33941)
0620 04:25:33.072244 (+ 632us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:33.072249 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:33.072264 (+ 15us) server_negotiation.cc:407] Connection header received
0620 04:25:33.072316 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:33.072319 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:33.072367 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:33.072460 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:25:33.072702 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:33.073391 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:33.074772 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:33.075344 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:33.076483 (+ 1139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:33.076550 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:33.076594 (+ 44us) server_negotiation.cc:299] Negotiation successful
0620 04:25:33.076662 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":66,"threads_started":1}
I20260620 04:25:33.076915 7541 tablet_service.cc:1467] Processing CreateTablet for tablet 4ee163af0bf64b29a9bc33df55a960c5 (DEFAULT_TABLE table=test-table [id=c3e9a1127998473cb79f45fe83381b0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:25:33.077168 7541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ee163af0bf64b29a9bc33df55a960c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:25:33.077240 7678 tablet_service.cc:1467] Processing CreateTablet for tablet 4ee163af0bf64b29a9bc33df55a960c5 (DEFAULT_TABLE table=test-table [id=c3e9a1127998473cb79f45fe83381b0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:25:33.077476 7678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ee163af0bf64b29a9bc33df55a960c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:25:33.079881 7838 tablet_bootstrap.cc:492] T 4ee163af0bf64b29a9bc33df55a960c5 P 66d7beaf7d8a415f8e5e1b457f1ec504: Bootstrap starting.
I20260620 04:25:33.080188 7837 tablet_bootstrap.cc:492] T 4ee163af0bf64b29a9bc33df55a960c5 P 449f41f8c68044e29bb5d65d0f0652ef: Bootstrap starting.
I20260620 04:25:33.080806 7838 tablet_bootstrap.cc:654] T 4ee163af0bf64b29a9bc33df55a960c5 P 66d7beaf7d8a415f8e5e1b457f1ec504: Neither blocks nor log segments found. Creating new log.
I20260620 04:25:33.080916 7837 tablet_bootstrap.cc:654] T 4ee163af0bf64b29a9bc33df55a960c5 P 449f41f8c68044e29bb5d65d0f0652ef: Neither blocks nor log segments found. Creating new log.
I20260620 04:25:33.081599 7836 tablet_bootstrap.cc:492] T 4ee163af0bf64b29a9bc33df55a960c5 P bde7e8cb28e54e59b808c1bb8439dbf1: Bootstrap starting.
I20260620 04:25:33.081960 7838 tablet_bootstrap.cc:492] T 4ee163af0bf64b29a9bc33df55a960c5 P 66d7beaf7d8a415f8e5e1b457f1ec504: No bootstrap required, opened a new log
I20260620 04:25:33.082020 7838 ts_tablet_manager.cc:1397] T 4ee163af0bf64b29a9bc33df55a960c5 P 66d7beaf7d8a415f8e5e1b457f1ec504: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 04:25:33.082046 7837 tablet_bootstrap.cc:492] T 4ee163af0bf64b29a9bc33df55a960c5 P 449f41f8c68044e29bb5d65d0f0652ef: No bootstrap required, opened a new log
I20260620 04:25:33.082140 7837 ts_tablet_manager.cc:1397] T 4ee163af0bf64b29a9bc33df55a960c5 P 449f41f8c68044e29bb5d65d0f0652ef: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 04:25:33.082335 7838 raft_consensus.cc:348] T 4ee163af0bf64b29a9bc33df55a960c5 P 66d7beaf7d8a415f8e5e1b457f1ec504 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "449f41f8c68044e29bb5d65d0f0652ef" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42601 } } peers { permanent_uuid: "66d7beaf7d8a415f8e5e1b457f1ec504" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33941 } } peers { permanent_uuid: "bde7e8cb28e54e59b808c1bb8439dbf1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38783 } }
I20260620 04:25:33.082415 7838 raft_consensus.cc:374] T 4ee163af0bf64b29a9bc33df55a960c5 P 66d7beaf7d8a415f8e5e1b457f1ec504 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:25:33.082434 7838 raft_consensus.cc:729] T 4ee163af0bf64b29a9bc33df55a960c5 P 66d7beaf7d8a415f8e5e1b457f1ec504 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66d7beaf7d8a415f8e5e1b457f1ec504, State: Initialized, Role: FOLLOWER
I20260620 04:25:33.082542 7836 tablet_bootstrap.cc:654] T 4ee163af0bf64b29a9bc33df55a960c5 P bde7e8cb28e54e59b808c1bb8439dbf1: Neither blocks nor log segments found. Creating new log.
I20260620 04:25:33.082538 7838 consensus_queue.cc:260] T 4ee163af0bf64b29a9bc33df55a960c5 P 66d7beaf7d8a415f8e5e1b457f1ec504 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "449f41f8c68044e29bb5d65d0f0652ef" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42601 } } peers { permanent_uuid: "66d7beaf7d8a415f8e5e1b457f1ec504" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33941 } } peers { permanent_uuid: "bde7e8cb28e54e59b808c1bb8439dbf1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38783 } }
I20260620 04:25:33.082612 7837 raft_consensus.cc:348] T 4ee163af0bf64b29a9bc33df55a960c5 P 449f41f8c68044e29bb5d65d0f0652ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "449f41f8c68044e29bb5d65d0f0652ef" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42601 } } peers { permanent_uuid: "66d7beaf7d8a415f8e5e1b457f1ec504" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33941 } } peers { permanent_uuid: "bde7e8cb28e54e59b808c1bb8439dbf1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38783 } }
I20260620 04:25:33.082723 7837 raft_consensus.cc:374] T 4ee163af0bf64b29a9bc33df55a960c5 P 449f41f8c68044e29bb5d65d0f0652ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:25:33.082757 7837 raft_consensus.cc:729] T 4ee163af0bf64b29a9bc33df55a960c5 P 449f41f8c68044e29bb5d65d0f0652ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 449f41f8c68044e29bb5d65d0f0652ef, State: Initialized, Role: FOLLOWER
I20260620 04:25:33.082851 7838 ts_tablet_manager.cc:1428] T 4ee163af0bf64b29a9bc33df55a960c5 P 66d7beaf7d8a415f8e5e1b457f1ec504: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 04:25:33.082917 7837 consensus_queue.cc:260] T 4ee163af0bf64b29a9bc33df55a960c5 P 449f41f8c68044e29bb5d65d0f0652ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "449f41f8c68044e29bb5d65d0f0652ef" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42601 } } peers { permanent_uuid: "66d7beaf7d8a415f8e5e1b457f1ec504" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33941 } } peers { permanent_uuid: "bde7e8cb28e54e59b808c1bb8439dbf1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38783 } }
I20260620 04:25:33.083487 7837 ts_tablet_manager.cc:1428] T 4ee163af0bf64b29a9bc33df55a960c5 P 449f41f8c68044e29bb5d65d0f0652ef: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 04:25:33.084842 7845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:33.083935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37461 (local address 127.0.98.1:33873)
0620 04:25:33.084718 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:33.084768 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":718,"thread_start_us":82,"threads_started":1}
I20260620 04:25:33.085525 7845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:33.085271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37461 (local address 127.0.98.1:57111)
0620 04:25:33.085417 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:33.085462 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260620 04:25:33.085747 7470 heartbeater.cc:503] Master 127.0.98.60:45713 was elected leader, sending a full tablet report...
I20260620 04:25:33.086375 7744 heartbeater.cc:503] Master 127.0.98.60:45713 was elected leader, sending a full tablet report...
I20260620 04:25:33.086984 7844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:33.083389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37461 (local address 127.0.98.3:45903)
0620 04:25:33.086827 (+ 3438us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:33.086888 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3366,"thread_start_us":91,"threads_started":1}
I20260620 04:25:33.087807 7846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:33.087452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37461 (local address 127.0.98.3:49953)
0620 04:25:33.087693 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:33.087736 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":104,"threads_started":1}
I20260620 04:25:33.088234 7836 tablet_bootstrap.cc:492] T 4ee163af0bf64b29a9bc33df55a960c5 P bde7e8cb28e54e59b808c1bb8439dbf1: No bootstrap required, opened a new log
I20260620 04:25:33.088325 7836 ts_tablet_manager.cc:1397] T 4ee163af0bf64b29a9bc33df55a960c5 P bde7e8cb28e54e59b808c1bb8439dbf1: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260620 04:25:33.088702 7836 raft_consensus.cc:348] T 4ee163af0bf64b29a9bc33df55a960c5 P bde7e8cb28e54e59b808c1bb8439dbf1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "449f41f8c68044e29bb5d65d0f0652ef" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42601 } } peers { permanent_uuid: "66d7beaf7d8a415f8e5e1b457f1ec504" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33941 } } peers { permanent_uuid: "bde7e8cb28e54e59b808c1bb8439dbf1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38783 } }
I20260620 04:25:33.088796 7836 raft_consensus.cc:374] T 4ee163af0bf64b29a9bc33df55a960c5 P bde7e8cb28e54e59b808c1bb8439dbf1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:25:33.088824 7836 raft_consensus.cc:729] T 4ee163af0bf64b29a9bc33df55a960c5 P bde7e8cb28e54e59b808c1bb8439dbf1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bde7e8cb28e54e59b808c1bb8439dbf1, State: Initialized, Role: FOLLOWER
I20260620 04:25:33.088935 7836 consensus_queue.cc:260] T 4ee163af0bf64b29a9bc33df55a960c5 P bde7e8cb28e54e59b808c1bb8439dbf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "449f41f8c68044e29bb5d65d0f0652ef" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42601 } } peers { permanent_uuid: "66d7beaf7d8a415f8e5e1b457f1ec504" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33941 } } peers { permanent_uuid: "bde7e8cb28e54e59b808c1bb8439dbf1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38783 } }
I20260620 04:25:33.089306 7836 ts_tablet_manager.cc:1428] T 4ee163af0bf64b29a9bc33df55a960c5 P bde7e8cb28e54e59b808c1bb8439dbf1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 04:25:33.090111 7848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:33.089705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37461 (local address 127.0.98.2:36079)
0620 04:25:33.089934 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:33.089991 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":98,"threads_started":1}
I20260620 04:25:33.090863 7607 heartbeater.cc:503] Master 127.0.98.60:45713 was elected leader, sending a full tablet report...
I20260620 04:25:33.090929 7848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:33.090632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37461 (local address 127.0.98.2:54143)
0620 04:25:33.090834 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:33.090875 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149}
W20260620 04:25:33.134270 7747 tablet.cc:2369] T 4ee163af0bf64b29a9bc33df55a960c5 P 66d7beaf7d8a415f8e5e1b457f1ec504: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 04:25:33.173506 7843 raft_consensus.cc:482] T 4ee163af0bf64b29a9bc33df55a960c5 P 449f41f8c68044e29bb5d65d0f0652ef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:25:33.173684 7843 raft_consensus.cc:504] T 4ee163af0bf64b29a9bc33df55a960c5 P 449f41f8c68044e29bb5d65d0f0652ef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "449f41f8c68044e29bb5d65d0f0652ef" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42601 } } peers { permanent_uuid: "66d7beaf7d8a415f8e5e1b457f1ec504" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33941 } } peers { permanent_uuid: "bde7e8cb28e54e59b808c1bb8439dbf1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38783 } }
I20260620 04:25:33.174767 7843 leader_election.cc:288] T 4ee163af0bf64b29a9bc33df55a960c5 P 449f41f8c68044e29bb5d65d0f0652ef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 66d7beaf7d8a415f8e5e1b457f1ec504 (127.0.98.3:33941), bde7e8cb28e54e59b808c1bb8439dbf1 (127.0.98.2:38783)
I20260620 04:25:33.180001 7835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:33.175112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48901 (local address 127.0.98.2:38783)
0620 04:25:33.175243 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:33.175247 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:33.175387 (+ 140us) server_negotiation.cc:407] Connection header received
0620 04:25:33.175481 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:33.175484 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:33.175528 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:33.175595 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:25:33.176066 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:33.176852 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:33.178625 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:33.179231 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:33.179339 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:33.179405 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:33.179756 (+ 351us) server_negotiation.cc:299] Negotiation successful
0620 04:25:33.179855 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 04:25:33.180032 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:33.175037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35683 (local address 127.0.98.3:33941)
0620 04:25:33.175162 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:33.175167 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:33.175238 (+ 71us) server_negotiation.cc:407] Connection header received
0620 04:25:33.175438 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:33.175441 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:33.175499 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:33.175587 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:25:33.176066 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:33.176852 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:33.178624 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:33.179241 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:33.179352 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:33.179405 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:33.179837 (+ 432us) server_negotiation.cc:299] Negotiation successful
0620 04:25:33.179913 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 04:25:33.180008 7845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:33.174973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33941 (local address 127.0.98.1:35683)
0620 04:25:33.175160 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:33.175175 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:33.175312 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:33.175644 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:33.175648 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:33.175706 (+ 58us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:25:33.175926 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:33.175931 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:33.176977 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:33.176982 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:33.178495 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:33.178504 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:33.179384 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:33.179388 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:33.179647 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:33.179667 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 04:25:33.179808 (+ 141us) client_negotiation.cc:240] Negotiation successful
0620 04:25:33.179860 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"mutex_wait_us":199,"spinlock_wait_cycles":63232}
I20260620 04:25:33.180346 7849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:33.175008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38783 (local address 127.0.98.1:48901)
0620 04:25:33.175308 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:33.175319 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:33.175397 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:33.175644 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:33.175648 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:33.175706 (+ 58us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:25:33.175920 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:33.175929 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:33.176977 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:33.176982 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:33.178492 (+ 1510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:33.178504 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:33.179376 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:33.179381 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:33.179578 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:33.179632 (+ 54us) client_negotiation.cc:769] Sending connection context
0620 04:25:33.179736 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 04:25:33.179817 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"mutex_wait_us":172,"spinlock_wait_cycles":6144,"thread_start_us":123,"threads_started":1}
I20260620 04:25:33.180490 7847 raft_consensus.cc:482] T 4ee163af0bf64b29a9bc33df55a960c5 P bde7e8cb28e54e59b808c1bb8439dbf1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:25:33.180594 7847 raft_consensus.cc:504] T 4ee163af0bf64b29a9bc33df55a960c5 P bde7e8cb28e54e59b808c1bb8439dbf1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "449f41f8c68044e29bb5d65d0f0652ef" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42601 } } peers { permanent_uuid: "66d7beaf7d8a415f8e5e1b457f1ec504" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33941 } } peers { permanent_uuid: "bde7e8cb28e54e59b808c1bb8439dbf1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38783 } }
I20260620 04:25:33.180825 7698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ee163af0bf64b29a9bc33df55a960c5" candidate_uuid: "449f41f8c68044e29bb5d65d0f0652ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66d7beaf7d8a415f8e5e1b457f1ec504" is_pre_election: true
I20260620 04:25:33.180970 7561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ee163af0bf64b29a9bc33df55a960c5" candidate_uuid: "449f41f8c68044e29bb5d65d0f0652ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bde7e8cb28e54e59b808c1bb8439dbf1" is_pre_election: true
I20260620 04:25:33.181056 7698 raft_consensus.cc:2454] T 4ee163af0bf64b29a9bc33df55a960c5 P 66d7beaf7d8a415f8e5e1b457f1ec504 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 449f41f8c68044e29bb5d65d0f0652ef in term 0.
I20260620 04:25:33.181082 7561 raft_consensus.cc:2454] T 4ee163af0bf64b29a9bc33df55a960c5 P bde7e8cb28e54e59b808c1bb8439dbf1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 449f41f8c68044e29bb5d65d0f0652ef in term 0.
I20260620 04:25:33.181443 7358 leader_election.cc:302] T 4ee163af0bf64b29a9bc33df55a960c5 P 449f41f8c68044e29bb5d65d0f0652ef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 449f41f8c68044e29bb5d65d0f0652ef, 66d7beaf7d8a415f8e5e1b457f1ec504; no voters:
I20260620 04:25:33.181695 7843 raft_consensus.cc:2781] T 4ee163af0bf64b29a9bc33df55a960c5 P 449f41f8c68044e29bb5d65d0f0652ef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 04:25:33.181753 7843 raft_consensus.cc:482] T 4ee163af0bf64b29a9bc33df55a960c5 P 449f41f8c68044e29bb5d65d0f0652ef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 04:25:33.181787 7843 raft_consensus.cc:3037] T 4ee163af0bf64b29a9bc33df55a960c5 P 449f41f8c68044e29bb5d65d0f0652ef [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:25:33.181841 7847 leader_election.cc:288] T 4ee163af0bf64b29a9bc33df55a960c5 P bde7e8cb28e54e59b808c1bb8439dbf1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 449f41f8c68044e29bb5d65d0f0652ef (127.0.98.1:42601), 66d7beaf7d8a415f8e5e1b457f1ec504 (127.0.98.3:33941)
I20260620 04:25:33.183089 7843 raft_consensus.cc:504] T 4ee163af0bf64b29a9bc33df55a960c5 P 449f41f8c68044e29bb5d65d0f0652ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "449f41f8c68044e29bb5d65d0f0652ef" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42601 } } peers { permanent_uuid: "66d7beaf7d8a415f8e5e1b457f1ec504" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33941 } } peers { permanent_uuid: "bde7e8cb28e54e59b808c1bb8439dbf1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38783 } }
I20260620 04:25:33.183482 7843 leader_election.cc:288] T 4ee163af0bf64b29a9bc33df55a960c5 P 449f41f8c68044e29bb5d65d0f0652ef [CANDIDATE]: Term 1 election: Requested vote from peers 66d7beaf7d8a415f8e5e1b457f1ec504 (127.0.98.3:33941), bde7e8cb28e54e59b808c1bb8439dbf1 (127.0.98.2:38783)
I20260620 04:25:33.183816 7561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ee163af0bf64b29a9bc33df55a960c5" candidate_uuid: "449f41f8c68044e29bb5d65d0f0652ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bde7e8cb28e54e59b808c1bb8439dbf1"
I20260620 04:25:33.183936 7561 raft_consensus.cc:3037] T 4ee163af0bf64b29a9bc33df55a960c5 P bde7e8cb28e54e59b808c1bb8439dbf1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:25:33.184288 7698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ee163af0bf64b29a9bc33df55a960c5" candidate_uuid: "449f41f8c68044e29bb5d65d0f0652ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66d7beaf7d8a415f8e5e1b457f1ec504"
I20260620 04:25:33.184401 7698 raft_consensus.cc:3037] T 4ee163af0bf64b29a9bc33df55a960c5 P 66d7beaf7d8a415f8e5e1b457f1ec504 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:25:33.185026 7561 raft_consensus.cc:2454] T 4ee163af0bf64b29a9bc33df55a960c5 P bde7e8cb28e54e59b808c1bb8439dbf1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 449f41f8c68044e29bb5d65d0f0652ef in term 1.
I20260620 04:25:33.185318 7359 leader_election.cc:302] T 4ee163af0bf64b29a9bc33df55a960c5 P 449f41f8c68044e29bb5d65d0f0652ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 449f41f8c68044e29bb5d65d0f0652ef, bde7e8cb28e54e59b808c1bb8439dbf1; no voters:
I20260620 04:25:33.185474 7698 raft_consensus.cc:2454] T 4ee163af0bf64b29a9bc33df55a960c5 P 66d7beaf7d8a415f8e5e1b457f1ec504 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 449f41f8c68044e29bb5d65d0f0652ef in term 1.
I20260620 04:25:33.185726 7843 raft_consensus.cc:2781] T 4ee163af0bf64b29a9bc33df55a960c5 P 449f41f8c68044e29bb5d65d0f0652ef [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:25:33.185967 7843 raft_consensus.cc:686] T 4ee163af0bf64b29a9bc33df55a960c5 P 449f41f8c68044e29bb5d65d0f0652ef [term 1 LEADER]: Becoming Leader. State: Replica: 449f41f8c68044e29bb5d65d0f0652ef, State: Running, Role: LEADER
I20260620 04:25:33.186456 7843 consensus_queue.cc:237] T 4ee163af0bf64b29a9bc33df55a960c5 P 449f41f8c68044e29bb5d65d0f0652ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "449f41f8c68044e29bb5d65d0f0652ef" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42601 } } peers { permanent_uuid: "66d7beaf7d8a415f8e5e1b457f1ec504" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33941 } } peers { permanent_uuid: "bde7e8cb28e54e59b808c1bb8439dbf1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38783 } }
I20260620 04:25:33.186540 7850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:33.182054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42601 (local address 127.0.98.2:45001)
0620 04:25:33.182523 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:33.182536 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:33.182626 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:33.182825 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:33.182827 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:33.182841 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:25:33.182935 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:33.182941 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:33.183788 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:33.183791 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:33.185023 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:33.185033 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:33.185957 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:33.185960 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:33.186195 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:33.186219 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 04:25:33.186327 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 04:25:33.186402 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":408,"mutex_wait_us":26}
I20260620 04:25:33.186540 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:33.182118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45001 (local address 127.0.98.1:42601)
0620 04:25:33.182221 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:33.182225 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:33.182604 (+ 379us) server_negotiation.cc:407] Connection header received
0620 04:25:33.182702 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:33.182705 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:33.182747 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:33.182809 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:25:33.183027 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:33.183683 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:33.185165 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:33.185753 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:33.185831 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:33.185891 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:33.186342 (+ 451us) server_negotiation.cc:299] Negotiation successful
0620 04:25:33.186404 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 04:25:33.186985 7848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:33.182272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33941 (local address 127.0.98.2:51919)
0620 04:25:33.182548 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:33.182561 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:33.182697 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:33.183010 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:33.183013 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:33.183029 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:25:33.183123 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:33.183129 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:33.184161 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:33.184164 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:33.185323 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:33.185334 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:33.186608 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:33.186611 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:33.186711 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:33.186732 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 04:25:33.186798 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 04:25:33.186859 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"mutex_wait_us":25,"thread_start_us":112,"threads_started":1}
I20260620 04:25:33.186605 7849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:33.186335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37461 (local address 127.0.98.1:48215)
0620 04:25:33.186471 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:33.186520 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260620 04:25:33.186964 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:33.182350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51919 (local address 127.0.98.3:33941)
0620 04:25:33.182446 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:33.182450 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:33.182631 (+ 181us) server_negotiation.cc:407] Connection header received
0620 04:25:33.182801 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:33.182805 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:33.182846 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:33.182890 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:25:33.183238 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:33.183899 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:33.185454 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:33.186008 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:33.186082 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:33.186134 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:33.186812 (+ 678us) server_negotiation.cc:299] Negotiation successful
0620 04:25:33.186869 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 04:25:33.187196 7424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ee163af0bf64b29a9bc33df55a960c5" candidate_uuid: "bde7e8cb28e54e59b808c1bb8439dbf1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "449f41f8c68044e29bb5d65d0f0652ef" is_pre_election: true
I20260620 04:25:33.187530 7698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ee163af0bf64b29a9bc33df55a960c5" candidate_uuid: "bde7e8cb28e54e59b808c1bb8439dbf1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66d7beaf7d8a415f8e5e1b457f1ec504" is_pre_election: true
I20260620 04:25:33.187702 7698 raft_consensus.cc:2379] T 4ee163af0bf64b29a9bc33df55a960c5 P 66d7beaf7d8a415f8e5e1b457f1ec504 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bde7e8cb28e54e59b808c1bb8439dbf1 in current term 1: Already voted for candidate 449f41f8c68044e29bb5d65d0f0652ef in this term.
I20260620 04:25:33.188097 7495 leader_election.cc:302] T 4ee163af0bf64b29a9bc33df55a960c5 P bde7e8cb28e54e59b808c1bb8439dbf1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bde7e8cb28e54e59b808c1bb8439dbf1; no voters: 449f41f8c68044e29bb5d65d0f0652ef, 66d7beaf7d8a415f8e5e1b457f1ec504
I20260620 04:25:33.188335 7847 raft_consensus.cc:2726] T 4ee163af0bf64b29a9bc33df55a960c5 P bde7e8cb28e54e59b808c1bb8439dbf1 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 04:25:33.190232 7291 catalog_manager.cc:5495] T 4ee163af0bf64b29a9bc33df55a960c5 P 449f41f8c68044e29bb5d65d0f0652ef reported cstate change: term changed from 0 to 1, leader changed from <none> to 449f41f8c68044e29bb5d65d0f0652ef (127.0.98.1). New cstate: current_term: 1 leader_uuid: "449f41f8c68044e29bb5d65d0f0652ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "449f41f8c68044e29bb5d65d0f0652ef" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66d7beaf7d8a415f8e5e1b457f1ec504" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bde7e8cb28e54e59b808c1bb8439dbf1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38783 } health_report { overall_health: UNKNOWN } } }
W20260620 04:25:33.194676 7610 tablet.cc:2369] T 4ee163af0bf64b29a9bc33df55a960c5 P bde7e8cb28e54e59b808c1bb8439dbf1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 04:25:33.230038 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:33.226554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59322 (local address 127.0.98.1:42601)
0620 04:25:33.226686 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:33.226691 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:33.226704 (+ 13us) server_negotiation.cc:407] Connection header received
0620 04:25:33.226784 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:33.226786 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:33.226835 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:33.226926 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:25:33.227144 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:33.227694 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:33.228639 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:33.228855 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:33.228943 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:33.229182 (+ 239us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:25:33.229667 (+ 485us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:25:33.229767 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:33.229840 (+ 73us) server_negotiation.cc:299] Negotiation successful
0620 04:25:33.229901 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 04:25:33.232892 7561 raft_consensus.cc:1261] T 4ee163af0bf64b29a9bc33df55a960c5 P bde7e8cb28e54e59b808c1bb8439dbf1 [term 1 FOLLOWER]: Refusing update from remote peer 449f41f8c68044e29bb5d65d0f0652ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:25:33.233047 7698 raft_consensus.cc:1261] T 4ee163af0bf64b29a9bc33df55a960c5 P 66d7beaf7d8a415f8e5e1b457f1ec504 [term 1 FOLLOWER]: Refusing update from remote peer 449f41f8c68044e29bb5d65d0f0652ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:25:33.233381 7851 consensus_queue.cc:1035] T 4ee163af0bf64b29a9bc33df55a960c5 P 449f41f8c68044e29bb5d65d0f0652ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "bde7e8cb28e54e59b808c1bb8439dbf1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:25:33.233536 7843 consensus_queue.cc:1035] T 4ee163af0bf64b29a9bc33df55a960c5 P 449f41f8c68044e29bb5d65d0f0652ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "66d7beaf7d8a415f8e5e1b457f1ec504" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:25:33.235225 7844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:33.234937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37461 (local address 127.0.98.3:33661)
0620 04:25:33.235088 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:33.235146 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260620 04:25:33.235291 7848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:33.234992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37461 (local address 127.0.98.2:41229)
0620 04:25:33.235155 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:33.235214 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260620 04:25:33.239506 7849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:33.239157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37461 (local address 127.0.98.1:42433)
0620 04:25:33.239375 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:33.239430 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154}
I20260620 04:25:33.240212 7844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:33.239503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37461 (local address 127.0.98.3:56735)
0620 04:25:33.240093 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:33.240145 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":524}
I20260620 04:25:33.241698 7848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:33.238703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37461 (local address 127.0.98.2:33759)
0620 04:25:33.241591 (+ 2888us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:33.241639 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2822}
I20260620 04:25:33.242318 7849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:33.242049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37461 (local address 127.0.98.1:55753)
0620 04:25:33.242187 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:33.242236 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260620 04:25:33.243003 7855 mvcc.cc:205] Tried to move back new op lower bound from 7298783368116404224 to 7298783367939215360. Current Snapshot: MvccSnapshot[applied={T|T < 7298783368116404224}]
I20260620 04:25:33.246768 7289 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44158:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:25:33.246923 7289 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44158:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:25:33.250358 7289 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3b026c99d08842f5a61f0a658e08398a: Sending DeleteTablet for 3 replicas of tablet 4ee163af0bf64b29a9bc33df55a960c5
I20260620 04:25:33.250978 7404 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ee163af0bf64b29a9bc33df55a960c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:25:33 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:59314
I20260620 04:25:33.251119 7541 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ee163af0bf64b29a9bc33df55a960c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:25:33 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:36534
I20260620 04:25:33.251251 7678 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ee163af0bf64b29a9bc33df55a960c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:25:33 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:60604
I20260620 04:25:33.251319 7864 tablet_replica.cc:330] T 4ee163af0bf64b29a9bc33df55a960c5 P 449f41f8c68044e29bb5d65d0f0652ef: stopping tablet replica
I20260620 04:25:33.251437 7864 raft_consensus.cc:2229] T 4ee163af0bf64b29a9bc33df55a960c5 P 449f41f8c68044e29bb5d65d0f0652ef [term 1 LEADER]: Raft consensus shutting down.
I20260620 04:25:33.251543 7866 tablet_replica.cc:330] T 4ee163af0bf64b29a9bc33df55a960c5 P 66d7beaf7d8a415f8e5e1b457f1ec504: stopping tablet replica
I20260620 04:25:33.251694 7866 raft_consensus.cc:2229] T 4ee163af0bf64b29a9bc33df55a960c5 P 66d7beaf7d8a415f8e5e1b457f1ec504 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:25:33.251742 7864 raft_consensus.cc:2258] T 4ee163af0bf64b29a9bc33df55a960c5 P 449f41f8c68044e29bb5d65d0f0652ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:25:33.251917 7866 raft_consensus.cc:2258] T 4ee163af0bf64b29a9bc33df55a960c5 P 66d7beaf7d8a415f8e5e1b457f1ec504 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:25:33.252455 7864 ts_tablet_manager.cc:1905] T 4ee163af0bf64b29a9bc33df55a960c5 P 449f41f8c68044e29bb5d65d0f0652ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:25:33.252660 7865 tablet_replica.cc:330] T 4ee163af0bf64b29a9bc33df55a960c5 P bde7e8cb28e54e59b808c1bb8439dbf1: stopping tablet replica
I20260620 04:25:33.252686 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 7252
I20260620 04:25:33.253924 7865 raft_consensus.cc:2229] T 4ee163af0bf64b29a9bc33df55a960c5 P bde7e8cb28e54e59b808c1bb8439dbf1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:25:33.254057 7865 raft_consensus.cc:2258] T 4ee163af0bf64b29a9bc33df55a960c5 P bde7e8cb28e54e59b808c1bb8439dbf1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:25:33.254688 7865 ts_tablet_manager.cc:1905] T 4ee163af0bf64b29a9bc33df55a960c5 P bde7e8cb28e54e59b808c1bb8439dbf1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:25:33.256673 7865 ts_tablet_manager.cc:1918] T 4ee163af0bf64b29a9bc33df55a960c5 P bde7e8cb28e54e59b808c1bb8439dbf1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:25:33.256734 7865 log.cc:1196] T 4ee163af0bf64b29a9bc33df55a960c5 P bde7e8cb28e54e59b808c1bb8439dbf1: Deleting WAL directory at /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/ts-1/wal/wals/4ee163af0bf64b29a9bc33df55a960c5
I20260620 04:25:33.256969 7865 ts_tablet_manager.cc:1939] T 4ee163af0bf64b29a9bc33df55a960c5 P bde7e8cb28e54e59b808c1bb8439dbf1: Deleting consensus metadata
I20260620 04:25:33.257565 7277 catalog_manager.cc:4841] TS bde7e8cb28e54e59b808c1bb8439dbf1 (127.0.98.2:38783): tablet 4ee163af0bf64b29a9bc33df55a960c5 (table test-table [id=c3e9a1127998473cb79f45fe83381b0f]) successfully deleted
I20260620 04:25:33.258421 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 7478
I20260620 04:25:33.258693 7866 ts_tablet_manager.cc:1905] T 4ee163af0bf64b29a9bc33df55a960c5 P 66d7beaf7d8a415f8e5e1b457f1ec504: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:25:33.259167 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:33.258543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42601 (local address 127.0.0.1:59336)
0620 04:25:33.259041 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:33.259092 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418}
W20260620 04:25:33.260525 7276 catalog_manager.cc:4584] TS 449f41f8c68044e29bb5d65d0f0652ef (127.0.98.1:42601): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4ee163af0bf64b29a9bc33df55a960c5: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42601: connect: Connection refused (error 111)
I20260620 04:25:33.260995 7866 ts_tablet_manager.cc:1918] T 4ee163af0bf64b29a9bc33df55a960c5 P 66d7beaf7d8a415f8e5e1b457f1ec504: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:25:33.261061 7866 log.cc:1196] T 4ee163af0bf64b29a9bc33df55a960c5 P 66d7beaf7d8a415f8e5e1b457f1ec504: Deleting WAL directory at /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929496330398-392-0/minicluster-data/ts-2/wal/wals/4ee163af0bf64b29a9bc33df55a960c5
I20260620 04:25:33.261307 7866 ts_tablet_manager.cc:1939] T 4ee163af0bf64b29a9bc33df55a960c5 P 66d7beaf7d8a415f8e5e1b457f1ec504: Deleting consensus metadata
I20260620 04:25:33.261888 7276 catalog_manager.cc:4841] TS 66d7beaf7d8a415f8e5e1b457f1ec504 (127.0.98.3:33941): tablet 4ee163af0bf64b29a9bc33df55a960c5 (table test-table [id=c3e9a1127998473cb79f45fe83381b0f]) successfully deleted
I20260620 04:25:33.263829 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 7615
I20260620 04:25:33.269356 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 6972
I20260620 04:25:33.274202 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 7080
2026-06-20T04:25:33Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5066 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929496330398-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:25:33 dist-test-slave-nk6d krb5kdc[7869](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:25:33 dist-test-slave-nk6d krb5kdc[7869](info): set up 2 sockets
Jun 20 04:25:33 dist-test-slave-nk6d krb5kdc[7869](info): commencing operation
krb5kdc: starting...
W20260620 04:25:34.232323 392 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929496330398-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929496330398-392-0/krb5kdc/test-user.keytab.
Jun 20 04:25:34 dist-test-slave-nk6d krb5kdc[7869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929534, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:25:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:25:34Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929496330398-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929496330398-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260620 04:25:34.342504 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929496330398-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929496330398-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929496330398-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38521
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929496330398-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38521
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:25:34.447606 7882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:25:34.447891 7882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:25:34.447949 7882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:25:34.451900 7882 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:25:34.451979 7882 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 04:25:34.452004 7882 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:25:34.452029 7882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:25:34.452059 7882 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:25:34.452088 7882 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:25:34.457240 7882 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929496330398-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929496330398-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38521
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929496330398-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38521
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7882
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929496330398-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:25:34.458218 7882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:25:34.459160 7882 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:25:34.466868 7887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:25:34.466915 7888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:25:34.466876 7890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:25:34.467013 7882 server_base.cc:1034] running on GCE node
I20260620 04:25:34.467698 7882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:25:34.468765 7882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:25:34.469928 7882 hybrid_clock.cc:648] HybridClock initialized: now 1781929534469901 us; error 39 us; skew 500 ppm
Jun 20 04:25:34 dist-test-slave-nk6d krb5kdc[7869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929534, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:25:34.472640 7882 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260620 04:25:34.473475 7882 webserver.cc:460] Webserver started at http://127.0.98.62:37241/ using document root <none> and password file <none>
I20260620 04:25:34.473992 7882 fs_manager.cc:359] Metadata directory not provided
I20260620 04:25:34.474054 7882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:25:34.474229 7882 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:25:34.476200 7882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929496330398-392-0/minicluster-data/master-0/data/instance:
uuid: "ac9ebe4a2398411f93639541ab97f9e7"
format_stamp: "Formatted at 2026-06-20 04:25:34 on dist-test-slave-nk6d"
server_key: "34e48f88959415fe5658d2bd8d5fa684"
server_key_iv: "e750a6981947a2e12c916394e5b598f7"
server_key_version: "encryptionkey@0"
I20260620 04:25:34.476725 7882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929496330398-392-0/minicluster-data/master-0/wal/instance:
uuid: "ac9ebe4a2398411f93639541ab97f9e7"
format_stamp: "Formatted at 2026-06-20 04:25:34 on dist-test-slave-nk6d"
server_key: "34e48f88959415fe5658d2bd8d5fa684"
server_key_iv: "e750a6981947a2e12c916394e5b598f7"
server_key_version: "encryptionkey@0"
I20260620 04:25:34.480278 7882 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:25:34.482610 7897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:34.483904 7882 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:25:34.484086 7882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929496330398-392-0/minicluster-data/master-0/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929496330398-392-0/minicluster-data/master-0/wal
uuid: "ac9ebe4a2398411f93639541ab97f9e7"
format_stamp: "Formatted at 2026-06-20 04:25:34 on dist-test-slave-nk6d"
server_key: "34e48f88959415fe5658d2bd8d5fa684"
server_key_iv: "e750a6981947a2e12c916394e5b598f7"
server_key_version: "encryptionkey@0"
I20260620 04:25:34.484181 7882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929496330398-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929496330398-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929496330398-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:25:34.508052 7882 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:25:34.511302 7882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:25:34.511477 7882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:25:34.519017 7882 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38521
I20260620 04:25:34.519042 7959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38521 every 8 connection(s)
I20260620 04:25:34.519948 7882 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:25:34.522739 7960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:25:34.528033 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 7882
I20260620 04:25:34.528147 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929496330398-392-0/minicluster-data/master-0/wal/instance
I20260620 04:25:34.528388 392 external_mini_cluster.cc:1428] Setting key 1ecea5a2bfbe3fd47c72f897a7758cae
I20260620 04:25:34.528363 7960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac9ebe4a2398411f93639541ab97f9e7: Bootstrap starting.
I20260620 04:25:34.530936 7960 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac9ebe4a2398411f93639541ab97f9e7: Neither blocks nor log segments found. Creating new log.
I20260620 04:25:34.531666 7960 log.cc:824] T 00000000000000000000000000000000 P ac9ebe4a2398411f93639541ab97f9e7: Log is configured to *not* fsync() on all Append() calls
I20260620 04:25:34.533519 7960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac9ebe4a2398411f93639541ab97f9e7: No bootstrap required, opened a new log
Jun 20 04:25:34 dist-test-slave-nk6d krb5kdc[7869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929534, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 04:25:34.537251 7960 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac9ebe4a2398411f93639541ab97f9e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac9ebe4a2398411f93639541ab97f9e7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38521 } }
I20260620 04:25:34.537493 7960 raft_consensus.cc:374] T 00000000000000000000000000000000 P ac9ebe4a2398411f93639541ab97f9e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:25:34.537564 7960 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac9ebe4a2398411f93639541ab97f9e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac9ebe4a2398411f93639541ab97f9e7, State: Initialized, Role: FOLLOWER
I20260620 04:25:34.538017 7960 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac9ebe4a2398411f93639541ab97f9e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac9ebe4a2398411f93639541ab97f9e7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38521 } }
I20260620 04:25:34.538170 7960 raft_consensus.cc:388] T 00000000000000000000000000000000 P ac9ebe4a2398411f93639541ab97f9e7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:25:34.538251 7960 raft_consensus.cc:482] T 00000000000000000000000000000000 P ac9ebe4a2398411f93639541ab97f9e7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:25:34.538355 7960 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac9ebe4a2398411f93639541ab97f9e7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:25:34.539192 7960 raft_consensus.cc:504] T 00000000000000000000000000000000 P ac9ebe4a2398411f93639541ab97f9e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac9ebe4a2398411f93639541ab97f9e7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38521 } }
I20260620 04:25:34.539525 7960 leader_election.cc:302] T 00000000000000000000000000000000 P ac9ebe4a2398411f93639541ab97f9e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac9ebe4a2398411f93639541ab97f9e7; no voters:
I20260620 04:25:34.539827 7960 leader_election.cc:288] T 00000000000000000000000000000000 P ac9ebe4a2398411f93639541ab97f9e7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:25:34.539888 7965 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ac9ebe4a2398411f93639541ab97f9e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:25:34.540099 7965 raft_consensus.cc:686] T 00000000000000000000000000000000 P ac9ebe4a2398411f93639541ab97f9e7 [term 1 LEADER]: Becoming Leader. State: Replica: ac9ebe4a2398411f93639541ab97f9e7, State: Running, Role: LEADER
I20260620 04:25:34.540388 7965 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac9ebe4a2398411f93639541ab97f9e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac9ebe4a2398411f93639541ab97f9e7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38521 } }
I20260620 04:25:34.541231 7960 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac9ebe4a2398411f93639541ab97f9e7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:25:34.541877 7966 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac9ebe4a2398411f93639541ab97f9e7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ac9ebe4a2398411f93639541ab97f9e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac9ebe4a2398411f93639541ab97f9e7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38521 } } }
I20260620 04:25:34.541981 7966 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac9ebe4a2398411f93639541ab97f9e7 [sys.catalog]: This master's current role is: LEADER
I20260620 04:25:34.542129 7963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:34.529666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35818 (local address 127.0.98.62:38521)
0620 04:25:34.530134 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:34.530149 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:34.530184 (+ 35us) server_negotiation.cc:407] Connection header received
0620 04:25:34.530796 (+ 612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:34.530812 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:34.531182 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:34.531521 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:34.532320 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:34.533258 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:34.534094 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:34.534544 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:34.534689 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:34.537225 (+ 2536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:34.537230 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:34.537243 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:25:34.537267 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:34.538984 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:34.539362 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:34.539368 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:34.539374 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:34.539488 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:34.539764 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:34.539768 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:34.539770 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:34.540310 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:34.540496 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:34.540766 (+ 270us) server_negotiation.cc:299] Negotiation successful
0620 04:25:34.541125 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":140,"threads_started":1}
I20260620 04:25:34.542452 7967 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac9ebe4a2398411f93639541ab97f9e7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac9ebe4a2398411f93639541ab97f9e7. Latest consensus state: current_term: 1 leader_uuid: "ac9ebe4a2398411f93639541ab97f9e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac9ebe4a2398411f93639541ab97f9e7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38521 } } }
I20260620 04:25:34.542574 7967 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac9ebe4a2398411f93639541ab97f9e7 [sys.catalog]: This master's current role is: LEADER
I20260620 04:25:34.542883 7972 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:25:34.546120 7972 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:25:34.551641 7972 catalog_manager.cc:1347] Generated new cluster ID: d58d9e43e15744e7861172c5b36ef6f0
I20260620 04:25:34.551724 7972 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:25:34.573331 7972 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:25:34.574651 7972 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:25:34.582183 7972 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ac9ebe4a2398411f93639541ab97f9e7: Generated new TSK 0
I20260620 04:25:34.582963 7972 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929496330398-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929496330398-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260620 04:25:34.608620 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929496330398-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929496330398-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929496330398-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929496330398-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38521
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929496330398-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:25:34.715286 7986 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:25:34.715548 7986 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:25:34.715595 7986 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:25:34.719610 7986 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:25:34.719672 7986 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:25:34.719709 7986 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 04:25:34.719751 7986 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 04:25:34.725003 7986 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929496330398-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929496330398-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929496330398-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929496330398-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:38521
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7986
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929496330398-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:25:34.726014 7986 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:25:34.727003 7986 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:25:34.736115 7991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:25:34.736155 7986 server_base.cc:1034] running on GCE node
W20260620 04:25:34.736119 7992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:25:34.736428 7994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:25:34.736789 7986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:25:34.737385 7986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:25:34.738554 7986 hybrid_clock.cc:648] HybridClock initialized: now 1781929534738537 us; error 29 us; skew 500 ppm
Jun 20 04:25:34 dist-test-slave-nk6d krb5kdc[7869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929534, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:25:34.741202 7986 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260620 04:25:34.742022 7986 webserver.cc:460] Webserver started at http://127.0.98.1:43767/ using document root <none> and password file <none>
I20260620 04:25:34.742550 7986 fs_manager.cc:359] Metadata directory not provided
I20260620 04:25:34.742617 7986 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:25:34.742784 7986 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:25:34.744807 7986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929496330398-392-0/minicluster-data/ts-0/data/instance:
uuid: "cc00f1f932a34415b055f21185d7d3f1"
format_stamp: "Formatted at 2026-06-20 04:25:34 on dist-test-slave-nk6d"
server_key: "7d66d1c25f02e1858b4a79e7851e330c"
server_key_iv: "48b3f21a233717f04fcabfeb7b88047d"
server_key_version: "encryptionkey@0"
I20260620 04:25:34.745292 7986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929496330398-392-0/minicluster-data/ts-0/wal/instance:
uuid: "cc00f1f932a34415b055f21185d7d3f1"
format_stamp: "Formatted at 2026-06-20 04:25:34 on dist-test-slave-nk6d"
server_key: "7d66d1c25f02e1858b4a79e7851e330c"
server_key_iv: "48b3f21a233717f04fcabfeb7b88047d"
server_key_version: "encryptionkey@0"
I20260620 04:25:34.748862 7986 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 04:25:34.751112 8001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:34.752362 7986 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 04:25:34.752450 7986 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929496330398-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929496330398-392-0/minicluster-data/ts-0/wal
uuid: "cc00f1f932a34415b055f21185d7d3f1"
format_stamp: "Formatted at 2026-06-20 04:25:34 on dist-test-slave-nk6d"
server_key: "7d66d1c25f02e1858b4a79e7851e330c"
server_key_iv: "48b3f21a233717f04fcabfeb7b88047d"
server_key_version: "encryptionkey@0"
I20260620 04:25:34.752544 7986 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929496330398-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929496330398-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929496330398-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:25:34.765044 7986 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:25:34.768000 7986 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:25:34.768157 7986 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:25:34.769452 7986 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:25:34.769495 7986 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:34.769533 7986 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:25:34.769562 7986 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 04:25:34 dist-test-slave-nk6d krb5kdc[7869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929534, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 04:25:34.779080 7986 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43319
I20260620 04:25:34.779114 8116 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43319 every 8 connection(s)
I20260620 04:25:34.780051 7986 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929496330398-392-0/minicluster-data/ts-0/data/info.pb
I20260620 04:25:34.782955 7963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:34.770181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:32999 (local address 127.0.98.62:38521)
0620 04:25:34.770288 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:34.770291 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:34.771393 (+ 1102us) server_negotiation.cc:407] Connection header received
0620 04:25:34.772492 (+ 1099us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:34.772496 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:34.772550 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:34.772624 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:34.773954 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:34.774740 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:34.776427 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:34.776674 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:34.776755 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:34.779528 (+ 2773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:34.779532 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:34.779534 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:25:34.779560 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:34.781057 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:34.781562 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:34.781566 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:34.781568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:34.781652 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:34.781946 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:34.781949 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:34.781951 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:34.782126 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:34.782206 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:34.782675 (+ 469us) server_negotiation.cc:299] Negotiation successful
0620 04:25:34.782790 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 04:25:34.783651 8010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:34.770700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38521 (local address 127.0.98.1:32999)
0620 04:25:34.771212 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:34.771253 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:34.772242 (+ 989us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:34.772821 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:34.772830 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:34.773216 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:34.773810 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:34.773831 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:34.774893 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:34.774897 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:34.776302 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:34.776314 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:34.776830 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:34.776834 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:34.777024 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:34.777623 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:34.777645 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:34.779363 (+ 1718us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:25:34.781150 (+ 1787us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:34.781155 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:34.781169 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:34.781471 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:34.781728 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:34.781732 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:34.781734 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:34.781863 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:34.782214 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:25:34.782220 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:25:34.782466 (+ 246us) client_negotiation.cc:769] Sending connection context
0620 04:25:34.782621 (+ 155us) client_negotiation.cc:240] Negotiation successful
0620 04:25:34.782906 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":130,"threads_started":1}
I20260620 04:25:34.784171 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 7986
I20260620 04:25:34.784281 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929496330398-392-0/minicluster-data/ts-0/wal/instance
I20260620 04:25:34.784612 392 external_mini_cluster.cc:1428] Setting key 574cfbe87528cbafa16053cdaf341926
I20260620 04:25:34.784998 8117 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38521
I20260620 04:25:34.785127 8117 heartbeater.cc:464] Registering TS with master...
I20260620 04:25:34.785621 8117 heartbeater.cc:511] Master 127.0.98.62:38521 requested a full tablet report, sending...
I20260620 04:25:34.788622 7913 ts_manager.cc:194] Registered new tserver with Master: cc00f1f932a34415b055f21185d7d3f1 (127.0.98.1:43319)
I20260620 04:25:34.790027 7913 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:32999
I20260620 04:25:34.793974 8123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:34.787030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38521 (local address 127.0.98.1:56947)
0620 04:25:34.787284 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:34.787299 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:34.787386 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:34.787623 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:34.787626 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:34.787820 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:34.787988 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:34.787995 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:34.789018 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:34.789020 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:34.789556 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:34.789564 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:34.789975 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:34.789979 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:34.790067 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:34.790517 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:34.790532 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:34.790872 (+ 340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:25:34.792724 (+ 1852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:34.792726 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:34.792728 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:34.792896 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:34.793149 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:34.793151 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:34.793152 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:34.793213 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:34.793577 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:25:34.793579 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:25:34.793649 (+ 70us) client_negotiation.cc:769] Sending connection context
0620 04:25:34.793726 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 04:25:34.793818 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":85,"threads_started":1}
I20260620 04:25:34.794056 7963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:34.787131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56947 (local address 127.0.98.62:38521)
0620 04:25:34.787244 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:34.787247 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:34.787363 (+ 116us) server_negotiation.cc:407] Connection header received
0620 04:25:34.787446 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:34.787449 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:34.787493 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:34.787584 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:34.788077 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:34.788887 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:34.789661 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:34.789877 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:34.789960 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:34.790990 (+ 1030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:34.790994 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:34.790996 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:25:34.791020 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:34.792626 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:34.792991 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:34.792995 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:34.792997 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:34.793077 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:34.793283 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:34.793286 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:34.793288 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:34.793486 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:34.793581 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:34.793796 (+ 215us) server_negotiation.cc:299] Negotiation successful
0620 04:25:34.793910 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929496330398-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929496330398-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260620 04:25:34.808936 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929496330398-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929496330398-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929496330398-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929496330398-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38521
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929496330398-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:25:34.915473 8126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:25:34.915730 8126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:25:34.915772 8126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:25:34.919710 8126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:25:34.919766 8126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:25:34.919803 8126 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 04:25:34.919863 8126 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 04:25:34.924957 8126 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929496330398-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929496330398-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929496330398-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929496330398-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:38521
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8126
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929496330398-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:25:34.925933 8126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:25:34.926900 8126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:25:34.936117 8132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:25:34.936198 8126 server_base.cc:1034] running on GCE node
W20260620 04:25:34.936143 8134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:25:34.936162 8131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:25:34.936705 8126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:25:34.937399 8126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:25:34.938580 8126 hybrid_clock.cc:648] HybridClock initialized: now 1781929534938560 us; error 29 us; skew 500 ppm
Jun 20 04:25:34 dist-test-slave-nk6d krb5kdc[7869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929534, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:25:34.941347 8126 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260620 04:25:34.942327 8126 webserver.cc:460] Webserver started at http://127.0.98.2:34717/ using document root <none> and password file <none>
I20260620 04:25:34.942952 8126 fs_manager.cc:359] Metadata directory not provided
I20260620 04:25:34.943040 8126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:25:34.943250 8126 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:25:34.945410 8126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929496330398-392-0/minicluster-data/ts-1/data/instance:
uuid: "f76fa9ea200d4526b6d05f551c0478f9"
format_stamp: "Formatted at 2026-06-20 04:25:34 on dist-test-slave-nk6d"
server_key: "d5732a210a38a29be79172081738e3be"
server_key_iv: "960dfd3905beb5187ee18187cb74581d"
server_key_version: "encryptionkey@0"
I20260620 04:25:34.945973 8126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929496330398-392-0/minicluster-data/ts-1/wal/instance:
uuid: "f76fa9ea200d4526b6d05f551c0478f9"
format_stamp: "Formatted at 2026-06-20 04:25:34 on dist-test-slave-nk6d"
server_key: "d5732a210a38a29be79172081738e3be"
server_key_iv: "960dfd3905beb5187ee18187cb74581d"
server_key_version: "encryptionkey@0"
I20260620 04:25:34.949757 8126 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 04:25:34.952169 8141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:34.953364 8126 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:25:34.953469 8126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929496330398-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929496330398-392-0/minicluster-data/ts-1/wal
uuid: "f76fa9ea200d4526b6d05f551c0478f9"
format_stamp: "Formatted at 2026-06-20 04:25:34 on dist-test-slave-nk6d"
server_key: "d5732a210a38a29be79172081738e3be"
server_key_iv: "960dfd3905beb5187ee18187cb74581d"
server_key_version: "encryptionkey@0"
I20260620 04:25:34.953570 8126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929496330398-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929496330398-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929496330398-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:25:34.967586 8126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:25:34.970422 8126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:25:34.970583 8126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:25:34.971915 8126 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:25:34.971985 8126 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:34.972038 8126 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:25:34.972062 8126 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 04:25:34 dist-test-slave-nk6d krb5kdc[7869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929534, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 04:25:34.981666 8126 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45413
I20260620 04:25:34.981711 8256 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45413 every 8 connection(s)
I20260620 04:25:34.982559 8126 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929496330398-392-0/minicluster-data/ts-1/data/info.pb
I20260620 04:25:34.984303 7963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:34.972747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49093 (local address 127.0.98.62:38521)
0620 04:25:34.972878 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:34.972883 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:34.973733 (+ 850us) server_negotiation.cc:407] Connection header received
0620 04:25:34.974922 (+ 1189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:34.974926 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:34.974982 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:34.975085 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:34.976615 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:34.977178 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:34.978378 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:34.978602 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:34.978689 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:34.981020 (+ 2331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:34.981024 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:34.981027 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:25:34.981051 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:34.982393 (+ 1342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:34.982872 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:34.982876 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:34.982878 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:34.982960 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:34.983294 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:34.983298 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:34.983300 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:34.983526 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:34.983615 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:34.984089 (+ 474us) server_negotiation.cc:299] Negotiation successful
0620 04:25:34.984185 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 04:25:34.984524 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 8126
I20260620 04:25:34.984673 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929496330398-392-0/minicluster-data/ts-1/wal/instance
I20260620 04:25:34.984946 392 external_mini_cluster.cc:1428] Setting key ff59000b201288b1cdbb58223d12c994
I20260620 04:25:34.985189 8150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:34.973113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38521 (local address 127.0.98.2:49093)
0620 04:25:34.973567 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:34.973622 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:34.974663 (+ 1041us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:34.975286 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:34.975296 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:34.975729 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:34.976441 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:34.976461 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:34.977304 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:34.977308 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:34.978267 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:34.978276 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:34.978745 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:34.978750 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:34.978899 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:34.979402 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:34.979417 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:34.980867 (+ 1450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:25:34.982495 (+ 1628us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:34.982500 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:34.982514 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:34.982784 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:34.983046 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:34.983049 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:34.983051 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:34.983211 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:34.983628 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:25:34.983634 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:25:34.983869 (+ 235us) client_negotiation.cc:769] Sending connection context
0620 04:25:34.984053 (+ 184us) client_negotiation.cc:240] Negotiation successful
0620 04:25:34.984315 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":91,"threads_started":1}
I20260620 04:25:34.986387 8257 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38521
I20260620 04:25:34.986526 8257 heartbeater.cc:464] Registering TS with master...
I20260620 04:25:34.987135 8257 heartbeater.cc:511] Master 127.0.98.62:38521 requested a full tablet report, sending...
I20260620 04:25:34.988368 7913 ts_manager.cc:194] Registered new tserver with Master: f76fa9ea200d4526b6d05f551c0478f9 (127.0.98.2:45413)
I20260620 04:25:34.989056 7913 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:49093
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 04:25:34.994711 7963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:34.988570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57369 (local address 127.0.98.62:38521)
0620 04:25:34.988662 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:34.988665 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:34.988677 (+ 12us) server_negotiation.cc:407] Connection header received
0620 04:25:34.988708 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:34.988711 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:34.988747 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:34.988823 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:34.989230 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:34.989831 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:34.990612 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:34.990827 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:34.990907 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:34.992002 (+ 1095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:34.992006 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:34.992009 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:25:34.992032 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:34.993277 (+ 1245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:34.993700 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:34.993702 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:34.993703 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:34.993765 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:34.994006 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:34.994008 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:34.994009 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:34.994184 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:34.994255 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:34.994512 (+ 257us) server_negotiation.cc:299] Negotiation successful
0620 04:25:34.994607 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 04:25:34.994784 8264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:34.988280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38521 (local address 127.0.98.2:57369)
0620 04:25:34.988542 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:34.988556 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:34.988638 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:34.988849 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:34.988852 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:34.989046 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:34.989149 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:34.989154 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:34.989953 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:34.989956 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:34.990515 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:34.990522 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:34.990920 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:34.990923 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:34.991007 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:34.991418 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:34.991432 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:34.991844 (+ 412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:25:34.993378 (+ 1534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:34.993382 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:34.993384 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:34.993618 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:34.993836 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:34.993839 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:34.993841 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:34.993930 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:34.994267 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:25:34.994270 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:25:34.994357 (+ 87us) client_negotiation.cc:769] Sending connection context
0620 04:25:34.994463 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 04:25:34.994602 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":88,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929496330398-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929496330398-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260620 04:25:35.009198 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929496330398-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929496330398-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929496330398-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929496330398-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38521
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929496330398-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:25:35.115746 8266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:25:35.116056 8266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:25:35.116104 8266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:25:35.120114 8266 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:25:35.120173 8266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:25:35.120210 8266 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 04:25:35.120247 8266 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 04:25:35.125402 8266 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929496330398-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929496330398-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929496330398-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929496330398-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:38521
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8266
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929496330398-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:25:35.126394 8266 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:25:35.127385 8266 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 04:25:35.136977 8266 server_base.cc:1034] running on GCE node
W20260620 04:25:35.136943 8271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:25:35.137046 8272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:25:35.137161 8274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:25:35.137610 8266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:25:35.138224 8266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:25:35.139377 8266 hybrid_clock.cc:648] HybridClock initialized: now 1781929535139353 us; error 37 us; skew 500 ppm
Jun 20 04:25:35 dist-test-slave-nk6d krb5kdc[7869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929535, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:25:35.141994 8266 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260620 04:25:35.142908 8266 webserver.cc:460] Webserver started at http://127.0.98.3:42461/ using document root <none> and password file <none>
I20260620 04:25:35.143427 8266 fs_manager.cc:359] Metadata directory not provided
I20260620 04:25:35.143484 8266 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:25:35.143673 8266 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:25:35.145673 8266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929496330398-392-0/minicluster-data/ts-2/data/instance:
uuid: "b6b68ffb3c80489f9a183bfcd20cafca"
format_stamp: "Formatted at 2026-06-20 04:25:35 on dist-test-slave-nk6d"
server_key: "c3882e8f3bf2f17f5f9e5305f783c632"
server_key_iv: "c06a4281aa2359aa90fb1c84bfc98a6c"
server_key_version: "encryptionkey@0"
I20260620 04:25:35.146186 8266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929496330398-392-0/minicluster-data/ts-2/wal/instance:
uuid: "b6b68ffb3c80489f9a183bfcd20cafca"
format_stamp: "Formatted at 2026-06-20 04:25:35 on dist-test-slave-nk6d"
server_key: "c3882e8f3bf2f17f5f9e5305f783c632"
server_key_iv: "c06a4281aa2359aa90fb1c84bfc98a6c"
server_key_version: "encryptionkey@0"
I20260620 04:25:35.149780 8266 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 04:25:35.152132 8281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:35.153285 8266 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:25:35.153374 8266 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929496330398-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929496330398-392-0/minicluster-data/ts-2/wal
uuid: "b6b68ffb3c80489f9a183bfcd20cafca"
format_stamp: "Formatted at 2026-06-20 04:25:35 on dist-test-slave-nk6d"
server_key: "c3882e8f3bf2f17f5f9e5305f783c632"
server_key_iv: "c06a4281aa2359aa90fb1c84bfc98a6c"
server_key_version: "encryptionkey@0"
I20260620 04:25:35.153470 8266 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929496330398-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929496330398-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929496330398-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:25:35.169894 8266 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:25:35.172780 8266 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:25:35.172941 8266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:25:35.174314 8266 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:25:35.174361 8266 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:35.174398 8266 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:25:35.174417 8266 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 04:25:35 dist-test-slave-nk6d krb5kdc[7869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929535, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 04:25:35.184098 8266 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42135
I20260620 04:25:35.184139 8396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42135 every 8 connection(s)
I20260620 04:25:35.185082 8266 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929496330398-392-0/minicluster-data/ts-2/data/info.pb
I20260620 04:25:35.187481 7963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:35.175124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47543 (local address 127.0.98.62:38521)
0620 04:25:35.175260 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:35.175265 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:35.176119 (+ 854us) server_negotiation.cc:407] Connection header received
0620 04:25:35.177163 (+ 1044us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:35.177167 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:35.177224 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:35.177327 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:35.178593 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:35.179114 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:35.180776 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:35.181064 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:35.181177 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:35.183875 (+ 2698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:35.183880 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:35.183883 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:25:35.183910 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:35.185578 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:35.186108 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:35.186112 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:35.186114 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:35.186200 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:35.186585 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:35.186587 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:35.186588 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:35.186747 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:35.186817 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:35.187217 (+ 400us) server_negotiation.cc:299] Negotiation successful
0620 04:25:35.187344 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 04:25:35.188272 8290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:35.175475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38521 (local address 127.0.98.3:47543)
0620 04:25:35.175933 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:35.176013 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:35.176950 (+ 937us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:35.177518 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:35.177528 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:35.177871 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:35.178458 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:35.178473 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:35.179228 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:35.179232 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:35.180643 (+ 1411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:35.180655 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:35.181224 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:35.181228 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:35.181412 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:35.181964 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:35.181984 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:35.183685 (+ 1701us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:25:35.185702 (+ 2017us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:35.185707 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:35.185719 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:35.186012 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:35.186346 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:35.186350 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:35.186352 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:35.186507 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:35.186818 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:25:35.186822 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:25:35.187031 (+ 209us) client_negotiation.cc:769] Sending connection context
0620 04:25:35.187177 (+ 146us) client_negotiation.cc:240] Negotiation successful
0620 04:25:35.187421 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":100,"threads_started":1}
I20260620 04:25:35.189323 8397 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38521
I20260620 04:25:35.189465 8397 heartbeater.cc:464] Registering TS with master...
I20260620 04:25:35.189899 8397 heartbeater.cc:511] Master 127.0.98.62:38521 requested a full tablet report, sending...
I20260620 04:25:35.190821 7913 ts_manager.cc:194] Registered new tserver with Master: b6b68ffb3c80489f9a183bfcd20cafca (127.0.98.3:42135)
I20260620 04:25:35.191537 7913 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47543
I20260620 04:25:35.194939 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 8266
I20260620 04:25:35.195050 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929496330398-392-0/minicluster-data/ts-2/wal/instance
I20260620 04:25:35.195336 392 external_mini_cluster.cc:1428] Setting key e9a204a511d8db5575b4792fdda9ec18
I20260620 04:25:35.197214 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 04:25:35.198064 7963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:35.191324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38835 (local address 127.0.98.62:38521)
0620 04:25:35.191419 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:35.191422 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:35.191921 (+ 499us) server_negotiation.cc:407] Connection header received
0620 04:25:35.192045 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:35.192048 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:35.192093 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:35.192152 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:35.192567 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:35.193036 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:35.193899 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:35.194099 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:35.194162 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:35.195236 (+ 1074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:35.195239 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:35.195241 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:25:35.195260 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:35.196620 (+ 1360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:35.196979 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:35.196982 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:35.196984 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:35.197068 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:35.197365 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:35.197368 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:35.197370 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:35.197523 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:35.197597 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:35.197838 (+ 241us) server_negotiation.cc:299] Negotiation successful
0620 04:25:35.197962 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 04:25:35.198098 8403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:35.191236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38521 (local address 127.0.98.3:38835)
0620 04:25:35.191850 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:35.191862 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:35.191948 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:35.192182 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:35.192185 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:35.192376 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:35.192479 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:35.192485 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:35.193144 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:35.193147 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:35.193812 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:35.193821 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:35.194176 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:35.194179 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:35.194246 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:35.194645 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:35.194674 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:35.195125 (+ 451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:25:35.196709 (+ 1584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:35.196711 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:35.196712 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:35.196898 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:35.197160 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:35.197163 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:35.197165 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:35.197255 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:35.197603 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:25:35.197606 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:25:35.197698 (+ 92us) client_negotiation.cc:769] Sending connection context
0620 04:25:35.197799 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 04:25:35.197927 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":100,"threads_started":1}
Jun 20 04:25:35 dist-test-slave-nk6d krb5kdc[7869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929535, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 04:25:35 dist-test-slave-nk6d krb5kdc[7869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:25:35 dist-test-slave-nk6d krb5kdc[7869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 04:25:35.218708 7963 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:25:35.213090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35834 (local address 127.0.98.62:38521)
0620 04:25:35.213221 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:35.213224 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:35.213235 (+ 11us) server_negotiation.cc:407] Connection header received
0620 04:25:35.213274 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:35.213277 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:35.213314 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:35.213382 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:35.214061 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:35.214563 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:35.215376 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:35.215585 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:35.215668 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:35.218578 (+ 2910us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35834: BlockingRecv error: recv got EOF from 127.0.0.1:35834 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jun 20 04:25:35 dist-test-slave-nk6d krb5kdc[7869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929535, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 04:25:35.227144 7963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:35.219998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35844 (local address 127.0.98.62:38521)
0620 04:25:35.220126 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:35.220130 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:35.220243 (+ 113us) server_negotiation.cc:407] Connection header received
0620 04:25:35.220327 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:35.220330 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:35.220366 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:35.220428 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:35.220912 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:35.221380 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:35.222066 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:35.222254 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:35.222318 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:35.224461 (+ 2143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:35.224464 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:35.224466 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:25:35.224486 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:35.225645 (+ 1159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:35.226078 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:35.226082 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:35.226083 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:35.226179 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:35.226424 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:35.226426 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:35.226428 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:35.226604 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:35.226707 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:35.226896 (+ 189us) server_negotiation.cc:299] Negotiation successful
0620 04:25:35.226991 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 04:25:35.230139 7913 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35844:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 04:25:35.233184 7913 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 04:25:35.247071 8421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:35.241581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60246 (local address 127.0.98.3:42135)
0620 04:25:35.242193 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:35.242205 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:35.242227 (+ 22us) server_negotiation.cc:407] Connection header received
0620 04:25:35.242298 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:35.242303 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:35.242513 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:35.242979 (+ 466us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:25:35.243113 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:35.244282 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:35.245697 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:35.246430 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:35.246525 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:35.246685 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:35.246835 (+ 150us) server_negotiation.cc:299] Negotiation successful
0620 04:25:35.246931 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":515,"thread_start_us":108,"threads_started":1}
I20260620 04:25:35.247061 8420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:35.241436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42135 (local address 127.0.0.1:60246)
0620 04:25:35.241957 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:35.242002 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:35.242129 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:35.242672 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:35.242678 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:35.242703 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:25:35.242835 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:35.242841 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:35.244397 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:35.244401 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:35.245457 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:35.245466 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:35.246543 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:35.246547 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:35.246660 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:35.246697 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 04:25:35.246799 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 04:25:35.246917 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"mutex_wait_us":27,"thread_start_us":73,"threads_started":1}
I20260620 04:25:35.247479 8422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:35.242166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45413 (local address 127.0.0.1:49960)
0620 04:25:35.242542 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:35.242555 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:35.242642 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:35.243571 (+ 929us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:35.243574 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:35.243589 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:25:35.243656 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:35.243660 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:35.244730 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:35.244732 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:35.245853 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:35.245863 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:35.247118 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:35.247121 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:35.247196 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:35.247217 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 04:25:35.247273 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 04:25:35.247346 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"mutex_wait_us":3,"thread_start_us":81,"threads_started":1}
I20260620 04:25:35.247488 8419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:35.241592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43319 (local address 127.0.0.1:33054)
0620 04:25:35.242319 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:35.242330 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:35.242391 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:35.243049 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:35.243052 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:35.243067 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:25:35.243162 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:35.243169 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:35.244581 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:35.244583 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:35.245906 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:35.245914 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:35.246943 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:35.246946 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:35.247042 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:35.247065 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 04:25:35.247113 (+ 48us) client_negotiation.cc:240] Negotiation successful
0620 04:25:35.247332 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":676,"mutex_wait_us":96,"thread_start_us":93,"threads_started":1}
I20260620 04:25:35.248049 8424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:35.242602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49960 (local address 127.0.98.2:45413)
0620 04:25:35.242986 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:35.243005 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:35.243031 (+ 26us) server_negotiation.cc:407] Connection header received
0620 04:25:35.243101 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:35.243106 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:35.243282 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:35.243430 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:25:35.243757 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:35.244636 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:35.245990 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:35.246814 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:35.246914 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:35.247066 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:35.247791 (+ 725us) server_negotiation.cc:299] Negotiation successful
0620 04:25:35.247880 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":91,"threads_started":1}
I20260620 04:25:35.248268 8423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:35.242131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33054 (local address 127.0.98.1:43319)
0620 04:25:35.242633 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:35.242645 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:35.242665 (+ 20us) server_negotiation.cc:407] Connection header received
0620 04:25:35.242728 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:35.242732 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:35.242913 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:35.243294 (+ 381us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:25:35.243422 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:35.244477 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:35.246018 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:35.246814 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:35.247840 (+ 1026us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:35.248011 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:35.248059 (+ 48us) server_negotiation.cc:299] Negotiation successful
0620 04:25:35.248143 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":102,"threads_started":1}
I20260620 04:25:35.250260 8331 tablet_service.cc:1467] Processing CreateTablet for tablet 8a43ff2c91c9427db4edb21f1ad1f151 (DEFAULT_TABLE table=test-table [id=8c82a51342fc48f4b2e8093e7c1f0fd6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:25:35.250578 8051 tablet_service.cc:1467] Processing CreateTablet for tablet 8a43ff2c91c9427db4edb21f1ad1f151 (DEFAULT_TABLE table=test-table [id=8c82a51342fc48f4b2e8093e7c1f0fd6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:25:35.250722 8191 tablet_service.cc:1467] Processing CreateTablet for tablet 8a43ff2c91c9427db4edb21f1ad1f151 (DEFAULT_TABLE table=test-table [id=8c82a51342fc48f4b2e8093e7c1f0fd6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:25:35.251408 8331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a43ff2c91c9427db4edb21f1ad1f151. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:25:35.251482 8051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a43ff2c91c9427db4edb21f1ad1f151. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:25:35.251583 8191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a43ff2c91c9427db4edb21f1ad1f151. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:25:35.258648 8426 tablet_bootstrap.cc:492] T 8a43ff2c91c9427db4edb21f1ad1f151 P b6b68ffb3c80489f9a183bfcd20cafca: Bootstrap starting.
I20260620 04:25:35.259388 8425 tablet_bootstrap.cc:492] T 8a43ff2c91c9427db4edb21f1ad1f151 P cc00f1f932a34415b055f21185d7d3f1: Bootstrap starting.
I20260620 04:25:35.260268 8427 tablet_bootstrap.cc:492] T 8a43ff2c91c9427db4edb21f1ad1f151 P f76fa9ea200d4526b6d05f551c0478f9: Bootstrap starting.
I20260620 04:25:35.261158 8426 tablet_bootstrap.cc:654] T 8a43ff2c91c9427db4edb21f1ad1f151 P b6b68ffb3c80489f9a183bfcd20cafca: Neither blocks nor log segments found. Creating new log.
I20260620 04:25:35.261312 8425 tablet_bootstrap.cc:654] T 8a43ff2c91c9427db4edb21f1ad1f151 P cc00f1f932a34415b055f21185d7d3f1: Neither blocks nor log segments found. Creating new log.
I20260620 04:25:35.261850 8427 tablet_bootstrap.cc:654] T 8a43ff2c91c9427db4edb21f1ad1f151 P f76fa9ea200d4526b6d05f551c0478f9: Neither blocks nor log segments found. Creating new log.
I20260620 04:25:35.262034 8426 log.cc:824] T 8a43ff2c91c9427db4edb21f1ad1f151 P b6b68ffb3c80489f9a183bfcd20cafca: Log is configured to *not* fsync() on all Append() calls
I20260620 04:25:35.262024 8425 log.cc:824] T 8a43ff2c91c9427db4edb21f1ad1f151 P cc00f1f932a34415b055f21185d7d3f1: Log is configured to *not* fsync() on all Append() calls
I20260620 04:25:35.262545 8427 log.cc:824] T 8a43ff2c91c9427db4edb21f1ad1f151 P f76fa9ea200d4526b6d05f551c0478f9: Log is configured to *not* fsync() on all Append() calls
I20260620 04:25:35.263922 8425 tablet_bootstrap.cc:492] T 8a43ff2c91c9427db4edb21f1ad1f151 P cc00f1f932a34415b055f21185d7d3f1: No bootstrap required, opened a new log
I20260620 04:25:35.264125 8425 ts_tablet_manager.cc:1397] T 8a43ff2c91c9427db4edb21f1ad1f151 P cc00f1f932a34415b055f21185d7d3f1: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260620 04:25:35.264556 8427 tablet_bootstrap.cc:492] T 8a43ff2c91c9427db4edb21f1ad1f151 P f76fa9ea200d4526b6d05f551c0478f9: No bootstrap required, opened a new log
I20260620 04:25:35.264705 8427 ts_tablet_manager.cc:1397] T 8a43ff2c91c9427db4edb21f1ad1f151 P f76fa9ea200d4526b6d05f551c0478f9: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 04:25:35.267103 8425 raft_consensus.cc:348] T 8a43ff2c91c9427db4edb21f1ad1f151 P cc00f1f932a34415b055f21185d7d3f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6b68ffb3c80489f9a183bfcd20cafca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42135 } } peers { permanent_uuid: "cc00f1f932a34415b055f21185d7d3f1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43319 } } peers { permanent_uuid: "f76fa9ea200d4526b6d05f551c0478f9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45413 } }
I20260620 04:25:35.267315 8425 raft_consensus.cc:374] T 8a43ff2c91c9427db4edb21f1ad1f151 P cc00f1f932a34415b055f21185d7d3f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:25:35.267367 8425 raft_consensus.cc:729] T 8a43ff2c91c9427db4edb21f1ad1f151 P cc00f1f932a34415b055f21185d7d3f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc00f1f932a34415b055f21185d7d3f1, State: Initialized, Role: FOLLOWER
I20260620 04:25:35.267464 8427 raft_consensus.cc:348] T 8a43ff2c91c9427db4edb21f1ad1f151 P f76fa9ea200d4526b6d05f551c0478f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6b68ffb3c80489f9a183bfcd20cafca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42135 } } peers { permanent_uuid: "cc00f1f932a34415b055f21185d7d3f1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43319 } } peers { permanent_uuid: "f76fa9ea200d4526b6d05f551c0478f9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45413 } }
I20260620 04:25:35.267733 8427 raft_consensus.cc:374] T 8a43ff2c91c9427db4edb21f1ad1f151 P f76fa9ea200d4526b6d05f551c0478f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:25:35.267791 8427 raft_consensus.cc:729] T 8a43ff2c91c9427db4edb21f1ad1f151 P f76fa9ea200d4526b6d05f551c0478f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f76fa9ea200d4526b6d05f551c0478f9, State: Initialized, Role: FOLLOWER
I20260620 04:25:35.267791 8425 consensus_queue.cc:260] T 8a43ff2c91c9427db4edb21f1ad1f151 P cc00f1f932a34415b055f21185d7d3f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6b68ffb3c80489f9a183bfcd20cafca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42135 } } peers { permanent_uuid: "cc00f1f932a34415b055f21185d7d3f1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43319 } } peers { permanent_uuid: "f76fa9ea200d4526b6d05f551c0478f9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45413 } }
I20260620 04:25:35.268183 8427 consensus_queue.cc:260] T 8a43ff2c91c9427db4edb21f1ad1f151 P f76fa9ea200d4526b6d05f551c0478f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6b68ffb3c80489f9a183bfcd20cafca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42135 } } peers { permanent_uuid: "cc00f1f932a34415b055f21185d7d3f1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43319 } } peers { permanent_uuid: "f76fa9ea200d4526b6d05f551c0478f9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45413 } }
I20260620 04:25:35.268476 8117 heartbeater.cc:503] Master 127.0.98.62:38521 was elected leader, sending a full tablet report...
I20260620 04:25:35.268766 8426 tablet_bootstrap.cc:492] T 8a43ff2c91c9427db4edb21f1ad1f151 P b6b68ffb3c80489f9a183bfcd20cafca: No bootstrap required, opened a new log
I20260620 04:25:35.268816 8425 ts_tablet_manager.cc:1428] T 8a43ff2c91c9427db4edb21f1ad1f151 P cc00f1f932a34415b055f21185d7d3f1: Time spent starting tablet: real 0.005s user 0.003s sys 0.002s
I20260620 04:25:35.268934 8426 ts_tablet_manager.cc:1397] T 8a43ff2c91c9427db4edb21f1ad1f151 P b6b68ffb3c80489f9a183bfcd20cafca: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20260620 04:25:35.269114 8427 ts_tablet_manager.cc:1428] T 8a43ff2c91c9427db4edb21f1ad1f151 P f76fa9ea200d4526b6d05f551c0478f9: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20260620 04:25:35.269451 8257 heartbeater.cc:503] Master 127.0.98.62:38521 was elected leader, sending a full tablet report...
I20260620 04:25:35.272586 8426 raft_consensus.cc:348] T 8a43ff2c91c9427db4edb21f1ad1f151 P b6b68ffb3c80489f9a183bfcd20cafca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6b68ffb3c80489f9a183bfcd20cafca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42135 } } peers { permanent_uuid: "cc00f1f932a34415b055f21185d7d3f1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43319 } } peers { permanent_uuid: "f76fa9ea200d4526b6d05f551c0478f9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45413 } }
I20260620 04:25:35.272841 8426 raft_consensus.cc:374] T 8a43ff2c91c9427db4edb21f1ad1f151 P b6b68ffb3c80489f9a183bfcd20cafca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:25:35.272902 8426 raft_consensus.cc:729] T 8a43ff2c91c9427db4edb21f1ad1f151 P b6b68ffb3c80489f9a183bfcd20cafca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6b68ffb3c80489f9a183bfcd20cafca, State: Initialized, Role: FOLLOWER
I20260620 04:25:35.273268 8426 consensus_queue.cc:260] T 8a43ff2c91c9427db4edb21f1ad1f151 P b6b68ffb3c80489f9a183bfcd20cafca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6b68ffb3c80489f9a183bfcd20cafca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42135 } } peers { permanent_uuid: "cc00f1f932a34415b055f21185d7d3f1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43319 } } peers { permanent_uuid: "f76fa9ea200d4526b6d05f551c0478f9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45413 } }
I20260620 04:25:35.273815 8397 heartbeater.cc:503] Master 127.0.98.62:38521 was elected leader, sending a full tablet report...
I20260620 04:25:35.274087 8426 ts_tablet_manager.cc:1428] T 8a43ff2c91c9427db4edb21f1ad1f151 P b6b68ffb3c80489f9a183bfcd20cafca: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260620 04:25:35.281194 8118 tablet.cc:2369] T 8a43ff2c91c9427db4edb21f1ad1f151 P cc00f1f932a34415b055f21185d7d3f1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 04:25:35.436280 8398 tablet.cc:2369] T 8a43ff2c91c9427db4edb21f1ad1f151 P b6b68ffb3c80489f9a183bfcd20cafca: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 04:25:35.483845 8258 tablet.cc:2369] T 8a43ff2c91c9427db4edb21f1ad1f151 P f76fa9ea200d4526b6d05f551c0478f9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 04:25:35.487463 8431 raft_consensus.cc:482] T 8a43ff2c91c9427db4edb21f1ad1f151 P cc00f1f932a34415b055f21185d7d3f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:25:35.487671 8431 raft_consensus.cc:504] T 8a43ff2c91c9427db4edb21f1ad1f151 P cc00f1f932a34415b055f21185d7d3f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6b68ffb3c80489f9a183bfcd20cafca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42135 } } peers { permanent_uuid: "cc00f1f932a34415b055f21185d7d3f1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43319 } } peers { permanent_uuid: "f76fa9ea200d4526b6d05f551c0478f9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45413 } }
I20260620 04:25:35.488790 8431 leader_election.cc:288] T 8a43ff2c91c9427db4edb21f1ad1f151 P cc00f1f932a34415b055f21185d7d3f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b6b68ffb3c80489f9a183bfcd20cafca (127.0.98.3:42135), f76fa9ea200d4526b6d05f551c0478f9 (127.0.98.2:45413)
I20260620 04:25:35.493996 8421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:35.489128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54615 (local address 127.0.98.3:42135)
0620 04:25:35.489258 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:35.489263 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:35.489458 (+ 195us) server_negotiation.cc:407] Connection header received
0620 04:25:35.489587 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:35.489590 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:35.489633 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:35.489716 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:25:35.490111 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:35.490899 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:35.492698 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:35.493306 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:35.493407 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:35.493469 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:35.493802 (+ 333us) server_negotiation.cc:299] Negotiation successful
0620 04:25:35.493869 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 04:25:35.493996 8424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:35.489096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52905 (local address 127.0.98.2:45413)
0620 04:25:35.489213 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:35.489218 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:35.489428 (+ 210us) server_negotiation.cc:407] Connection header received
0620 04:25:35.489575 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:35.489578 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:35.489633 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:35.489719 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:25:35.490138 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:35.490918 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:35.492698 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:35.493304 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:35.493407 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:35.493469 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:35.493802 (+ 333us) server_negotiation.cc:299] Negotiation successful
0620 04:25:35.493870 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 04:25:35.494083 8435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:35.489016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45413 (local address 127.0.98.1:52905)
0620 04:25:35.489336 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:35.489349 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:35.489469 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:35.489746 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:35.489749 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:35.489767 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:25:35.490032 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:35.490039 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:35.491021 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:35.491025 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:35.492563 (+ 1538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:35.492574 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:35.493430 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:35.493435 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:35.493650 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:35.493708 (+ 58us) client_negotiation.cc:769] Sending connection context
0620 04:25:35.493784 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 04:25:35.493904 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"mutex_wait_us":316,"spinlock_wait_cycles":6784,"thread_start_us":155,"threads_started":1}
I20260620 04:25:35.494282 8434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:35.489040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42135 (local address 127.0.98.1:54615)
0620 04:25:35.489383 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:35.489394 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:35.489500 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:35.489772 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:35.489774 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:35.489791 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:25:35.489983 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:35.489991 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:35.491021 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:35.491025 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:35.492563 (+ 1538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:35.492574 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:35.493431 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:35.493435 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:35.493653 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:35.493708 (+ 55us) client_negotiation.cc:769] Sending connection context
0620 04:25:35.493785 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 04:25:35.493906 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"mutex_wait_us":225,"spinlock_wait_cycles":1152,"thread_start_us":63,"threads_started":1}
I20260620 04:25:35.494812 8351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a43ff2c91c9427db4edb21f1ad1f151" candidate_uuid: "cc00f1f932a34415b055f21185d7d3f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6b68ffb3c80489f9a183bfcd20cafca" is_pre_election: true
I20260620 04:25:35.494789 8211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a43ff2c91c9427db4edb21f1ad1f151" candidate_uuid: "cc00f1f932a34415b055f21185d7d3f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f76fa9ea200d4526b6d05f551c0478f9" is_pre_election: true
I20260620 04:25:35.495118 8211 raft_consensus.cc:2454] T 8a43ff2c91c9427db4edb21f1ad1f151 P f76fa9ea200d4526b6d05f551c0478f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc00f1f932a34415b055f21185d7d3f1 in term 0.
I20260620 04:25:35.495119 8351 raft_consensus.cc:2454] T 8a43ff2c91c9427db4edb21f1ad1f151 P b6b68ffb3c80489f9a183bfcd20cafca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc00f1f932a34415b055f21185d7d3f1 in term 0.
I20260620 04:25:35.495631 8005 leader_election.cc:302] T 8a43ff2c91c9427db4edb21f1ad1f151 P cc00f1f932a34415b055f21185d7d3f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: b6b68ffb3c80489f9a183bfcd20cafca, cc00f1f932a34415b055f21185d7d3f1, f76fa9ea200d4526b6d05f551c0478f9; no voters:
I20260620 04:25:35.495887 8431 raft_consensus.cc:2781] T 8a43ff2c91c9427db4edb21f1ad1f151 P cc00f1f932a34415b055f21185d7d3f1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 04:25:35.495988 8431 raft_consensus.cc:482] T 8a43ff2c91c9427db4edb21f1ad1f151 P cc00f1f932a34415b055f21185d7d3f1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 04:25:35.496031 8431 raft_consensus.cc:3037] T 8a43ff2c91c9427db4edb21f1ad1f151 P cc00f1f932a34415b055f21185d7d3f1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:25:35.496994 8431 raft_consensus.cc:504] T 8a43ff2c91c9427db4edb21f1ad1f151 P cc00f1f932a34415b055f21185d7d3f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6b68ffb3c80489f9a183bfcd20cafca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42135 } } peers { permanent_uuid: "cc00f1f932a34415b055f21185d7d3f1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43319 } } peers { permanent_uuid: "f76fa9ea200d4526b6d05f551c0478f9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45413 } }
I20260620 04:25:35.497339 8431 leader_election.cc:288] T 8a43ff2c91c9427db4edb21f1ad1f151 P cc00f1f932a34415b055f21185d7d3f1 [CANDIDATE]: Term 1 election: Requested vote from peers b6b68ffb3c80489f9a183bfcd20cafca (127.0.98.3:42135), f76fa9ea200d4526b6d05f551c0478f9 (127.0.98.2:45413)
I20260620 04:25:35.497640 8351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a43ff2c91c9427db4edb21f1ad1f151" candidate_uuid: "cc00f1f932a34415b055f21185d7d3f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6b68ffb3c80489f9a183bfcd20cafca"
I20260620 04:25:35.497640 8211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a43ff2c91c9427db4edb21f1ad1f151" candidate_uuid: "cc00f1f932a34415b055f21185d7d3f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f76fa9ea200d4526b6d05f551c0478f9"
I20260620 04:25:35.497762 8211 raft_consensus.cc:3037] T 8a43ff2c91c9427db4edb21f1ad1f151 P f76fa9ea200d4526b6d05f551c0478f9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:25:35.497762 8351 raft_consensus.cc:3037] T 8a43ff2c91c9427db4edb21f1ad1f151 P b6b68ffb3c80489f9a183bfcd20cafca [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:25:35.499028 8211 raft_consensus.cc:2454] T 8a43ff2c91c9427db4edb21f1ad1f151 P f76fa9ea200d4526b6d05f551c0478f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc00f1f932a34415b055f21185d7d3f1 in term 1.
I20260620 04:25:35.499255 8351 raft_consensus.cc:2454] T 8a43ff2c91c9427db4edb21f1ad1f151 P b6b68ffb3c80489f9a183bfcd20cafca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc00f1f932a34415b055f21185d7d3f1 in term 1.
I20260620 04:25:35.499318 8005 leader_election.cc:302] T 8a43ff2c91c9427db4edb21f1ad1f151 P cc00f1f932a34415b055f21185d7d3f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cc00f1f932a34415b055f21185d7d3f1, f76fa9ea200d4526b6d05f551c0478f9; no voters:
I20260620 04:25:35.499537 8431 raft_consensus.cc:2781] T 8a43ff2c91c9427db4edb21f1ad1f151 P cc00f1f932a34415b055f21185d7d3f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:25:35.499707 8431 raft_consensus.cc:686] T 8a43ff2c91c9427db4edb21f1ad1f151 P cc00f1f932a34415b055f21185d7d3f1 [term 1 LEADER]: Becoming Leader. State: Replica: cc00f1f932a34415b055f21185d7d3f1, State: Running, Role: LEADER
I20260620 04:25:35.500068 8431 consensus_queue.cc:237] T 8a43ff2c91c9427db4edb21f1ad1f151 P cc00f1f932a34415b055f21185d7d3f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6b68ffb3c80489f9a183bfcd20cafca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42135 } } peers { permanent_uuid: "cc00f1f932a34415b055f21185d7d3f1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43319 } } peers { permanent_uuid: "f76fa9ea200d4526b6d05f551c0478f9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45413 } }
I20260620 04:25:35.503113 7913 catalog_manager.cc:5495] T 8a43ff2c91c9427db4edb21f1ad1f151 P cc00f1f932a34415b055f21185d7d3f1 reported cstate change: term changed from 0 to 1, leader changed from <none> to cc00f1f932a34415b055f21185d7d3f1 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "cc00f1f932a34415b055f21185d7d3f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6b68ffb3c80489f9a183bfcd20cafca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42135 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc00f1f932a34415b055f21185d7d3f1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f76fa9ea200d4526b6d05f551c0478f9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45413 } health_report { overall_health: UNKNOWN } } }
I20260620 04:25:35.538276 8423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:35.534970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33068 (local address 127.0.98.1:43319)
0620 04:25:35.535079 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:35.535083 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:35.535096 (+ 13us) server_negotiation.cc:407] Connection header received
0620 04:25:35.535253 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:35.535255 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:35.535311 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:35.535405 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:25:35.535635 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:35.536214 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:35.537031 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:35.537245 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:35.537323 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:35.537524 (+ 201us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:25:35.537954 (+ 430us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:25:35.538062 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:35.538118 (+ 56us) server_negotiation.cc:299] Negotiation successful
0620 04:25:35.538169 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 04:25:35.544127 8211 raft_consensus.cc:1261] T 8a43ff2c91c9427db4edb21f1ad1f151 P f76fa9ea200d4526b6d05f551c0478f9 [term 1 FOLLOWER]: Refusing update from remote peer cc00f1f932a34415b055f21185d7d3f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:25:35.544910 8436 consensus_queue.cc:1035] T 8a43ff2c91c9427db4edb21f1ad1f151 P cc00f1f932a34415b055f21185d7d3f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f76fa9ea200d4526b6d05f551c0478f9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:25:35.544996 8351 raft_consensus.cc:1261] T 8a43ff2c91c9427db4edb21f1ad1f151 P b6b68ffb3c80489f9a183bfcd20cafca [term 1 FOLLOWER]: Refusing update from remote peer cc00f1f932a34415b055f21185d7d3f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:25:35.546082 8436 consensus_queue.cc:1035] T 8a43ff2c91c9427db4edb21f1ad1f151 P cc00f1f932a34415b055f21185d7d3f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6b68ffb3c80489f9a183bfcd20cafca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:25:35.554486 8441 mvcc.cc:205] Tried to move back new op lower bound from 7298783377581105152 to 7298783377410600960. Current Snapshot: MvccSnapshot[applied={T|T < 7298783377581105152}]
I20260620 04:25:35.561586 7913 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35844:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:25:35.561741 7913 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35844:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:25:35.564102 7913 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ac9ebe4a2398411f93639541ab97f9e7: Sending DeleteTablet for 3 replicas of tablet 8a43ff2c91c9427db4edb21f1ad1f151
I20260620 04:25:35.564901 8051 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a43ff2c91c9427db4edb21f1ad1f151 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:25:35 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33054
I20260620 04:25:35.564944 8331 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a43ff2c91c9427db4edb21f1ad1f151 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:25:35 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60246
I20260620 04:25:35.565546 8448 tablet_replica.cc:330] T 8a43ff2c91c9427db4edb21f1ad1f151 P b6b68ffb3c80489f9a183bfcd20cafca: stopping tablet replica
I20260620 04:25:35.565529 7912 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35844:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 04:25:35.565603 8191 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a43ff2c91c9427db4edb21f1ad1f151 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:25:35 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49960
I20260620 04:25:35.565864 8448 raft_consensus.cc:2229] T 8a43ff2c91c9427db4edb21f1ad1f151 P b6b68ffb3c80489f9a183bfcd20cafca [term 1 FOLLOWER]: Raft consensus shutting down.
W20260620 04:25:35.565944 7912 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 04:25:35.566161 8448 raft_consensus.cc:2258] T 8a43ff2c91c9427db4edb21f1ad1f151 P b6b68ffb3c80489f9a183bfcd20cafca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:25:35.566403 8450 tablet_replica.cc:330] T 8a43ff2c91c9427db4edb21f1ad1f151 P f76fa9ea200d4526b6d05f551c0478f9: stopping tablet replica
I20260620 04:25:35.566774 8450 raft_consensus.cc:2229] T 8a43ff2c91c9427db4edb21f1ad1f151 P f76fa9ea200d4526b6d05f551c0478f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:25:35.567126 8450 raft_consensus.cc:2258] T 8a43ff2c91c9427db4edb21f1ad1f151 P f76fa9ea200d4526b6d05f551c0478f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:25:35.567767 8448 ts_tablet_manager.cc:1905] T 8a43ff2c91c9427db4edb21f1ad1f151 P b6b68ffb3c80489f9a183bfcd20cafca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:25:35.570422 8448 ts_tablet_manager.cc:1918] T 8a43ff2c91c9427db4edb21f1ad1f151 P b6b68ffb3c80489f9a183bfcd20cafca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:25:35.570513 8448 log.cc:1196] T 8a43ff2c91c9427db4edb21f1ad1f151 P b6b68ffb3c80489f9a183bfcd20cafca: Deleting WAL directory at /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929496330398-392-0/minicluster-data/ts-2/wal/wals/8a43ff2c91c9427db4edb21f1ad1f151
I20260620 04:25:35.570811 8448 ts_tablet_manager.cc:1939] T 8a43ff2c91c9427db4edb21f1ad1f151 P b6b68ffb3c80489f9a183bfcd20cafca: Deleting consensus metadata
I20260620 04:25:35.570886 8191 tablet_service.cc:1467] Processing CreateTablet for tablet 1263a5d7a46842dbaeb7e3f78a40c613 (DEFAULT_TABLE table=test-table [id=b916be3080cf40a2a6d1f91be609e5d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:25:35.571017 8331 tablet_service.cc:1467] Processing CreateTablet for tablet 1263a5d7a46842dbaeb7e3f78a40c613 (DEFAULT_TABLE table=test-table [id=b916be3080cf40a2a6d1f91be609e5d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:25:35.571141 8191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1263a5d7a46842dbaeb7e3f78a40c613. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:25:35.571209 8331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1263a5d7a46842dbaeb7e3f78a40c613. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:25:35.572026 8449 tablet_replica.cc:330] T 8a43ff2c91c9427db4edb21f1ad1f151 P cc00f1f932a34415b055f21185d7d3f1: stopping tablet replica
I20260620 04:25:35.572304 8449 raft_consensus.cc:2229] T 8a43ff2c91c9427db4edb21f1ad1f151 P cc00f1f932a34415b055f21185d7d3f1 [term 1 LEADER]: Raft consensus shutting down.
I20260620 04:25:35.572458 7900 catalog_manager.cc:4841] TS b6b68ffb3c80489f9a183bfcd20cafca (127.0.98.3:42135): tablet 8a43ff2c91c9427db4edb21f1ad1f151 (table test-table [id=8c82a51342fc48f4b2e8093e7c1f0fd6]) successfully deleted
I20260620 04:25:35.572674 8449 raft_consensus.cc:2258] T 8a43ff2c91c9427db4edb21f1ad1f151 P cc00f1f932a34415b055f21185d7d3f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:25:35.573405 8051 tablet_service.cc:1467] Processing CreateTablet for tablet 1263a5d7a46842dbaeb7e3f78a40c613 (DEFAULT_TABLE table=test-table [id=b916be3080cf40a2a6d1f91be609e5d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:25:35.573524 8427 tablet_bootstrap.cc:492] T 1263a5d7a46842dbaeb7e3f78a40c613 P f76fa9ea200d4526b6d05f551c0478f9: Bootstrap starting.
I20260620 04:25:35.573652 8051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1263a5d7a46842dbaeb7e3f78a40c613. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:25:35.573900 8426 tablet_bootstrap.cc:492] T 1263a5d7a46842dbaeb7e3f78a40c613 P b6b68ffb3c80489f9a183bfcd20cafca: Bootstrap starting.
I20260620 04:25:35.574437 8427 tablet_bootstrap.cc:654] T 1263a5d7a46842dbaeb7e3f78a40c613 P f76fa9ea200d4526b6d05f551c0478f9: Neither blocks nor log segments found. Creating new log.
I20260620 04:25:35.574739 8426 tablet_bootstrap.cc:654] T 1263a5d7a46842dbaeb7e3f78a40c613 P b6b68ffb3c80489f9a183bfcd20cafca: Neither blocks nor log segments found. Creating new log.
I20260620 04:25:35.575790 8426 tablet_bootstrap.cc:492] T 1263a5d7a46842dbaeb7e3f78a40c613 P b6b68ffb3c80489f9a183bfcd20cafca: No bootstrap required, opened a new log
I20260620 04:25:35.575855 8425 tablet_bootstrap.cc:492] T 1263a5d7a46842dbaeb7e3f78a40c613 P cc00f1f932a34415b055f21185d7d3f1: Bootstrap starting.
I20260620 04:25:35.575863 8426 ts_tablet_manager.cc:1397] T 1263a5d7a46842dbaeb7e3f78a40c613 P b6b68ffb3c80489f9a183bfcd20cafca: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 04:25:35.576304 8426 raft_consensus.cc:348] T 1263a5d7a46842dbaeb7e3f78a40c613 P b6b68ffb3c80489f9a183bfcd20cafca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc00f1f932a34415b055f21185d7d3f1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43319 } } peers { permanent_uuid: "f76fa9ea200d4526b6d05f551c0478f9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45413 } } peers { permanent_uuid: "b6b68ffb3c80489f9a183bfcd20cafca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42135 } }
I20260620 04:25:35.576402 8426 raft_consensus.cc:374] T 1263a5d7a46842dbaeb7e3f78a40c613 P b6b68ffb3c80489f9a183bfcd20cafca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:25:35.576431 8426 raft_consensus.cc:729] T 1263a5d7a46842dbaeb7e3f78a40c613 P b6b68ffb3c80489f9a183bfcd20cafca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6b68ffb3c80489f9a183bfcd20cafca, State: Initialized, Role: FOLLOWER
I20260620 04:25:35.576542 8426 consensus_queue.cc:260] T 1263a5d7a46842dbaeb7e3f78a40c613 P b6b68ffb3c80489f9a183bfcd20cafca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc00f1f932a34415b055f21185d7d3f1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43319 } } peers { permanent_uuid: "f76fa9ea200d4526b6d05f551c0478f9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45413 } } peers { permanent_uuid: "b6b68ffb3c80489f9a183bfcd20cafca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42135 } }
I20260620 04:25:35.576761 8426 ts_tablet_manager.cc:1428] T 1263a5d7a46842dbaeb7e3f78a40c613 P b6b68ffb3c80489f9a183bfcd20cafca: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 04:25:35.576782 8425 tablet_bootstrap.cc:654] T 1263a5d7a46842dbaeb7e3f78a40c613 P cc00f1f932a34415b055f21185d7d3f1: Neither blocks nor log segments found. Creating new log.
I20260620 04:25:35.576793 8450 ts_tablet_manager.cc:1905] T 8a43ff2c91c9427db4edb21f1ad1f151 P f76fa9ea200d4526b6d05f551c0478f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:25:35.578830 8449 ts_tablet_manager.cc:1905] T 8a43ff2c91c9427db4edb21f1ad1f151 P cc00f1f932a34415b055f21185d7d3f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:25:35.579882 8450 ts_tablet_manager.cc:1918] T 8a43ff2c91c9427db4edb21f1ad1f151 P f76fa9ea200d4526b6d05f551c0478f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:25:35.579993 8450 log.cc:1196] T 8a43ff2c91c9427db4edb21f1ad1f151 P f76fa9ea200d4526b6d05f551c0478f9: Deleting WAL directory at /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929496330398-392-0/minicluster-data/ts-1/wal/wals/8a43ff2c91c9427db4edb21f1ad1f151
I20260620 04:25:35.580637 8450 ts_tablet_manager.cc:1939] T 8a43ff2c91c9427db4edb21f1ad1f151 P f76fa9ea200d4526b6d05f551c0478f9: Deleting consensus metadata
I20260620 04:25:35.581200 8449 ts_tablet_manager.cc:1918] T 8a43ff2c91c9427db4edb21f1ad1f151 P cc00f1f932a34415b055f21185d7d3f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:25:35.581290 8449 log.cc:1196] T 8a43ff2c91c9427db4edb21f1ad1f151 P cc00f1f932a34415b055f21185d7d3f1: Deleting WAL directory at /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929496330398-392-0/minicluster-data/ts-0/wal/wals/8a43ff2c91c9427db4edb21f1ad1f151
I20260620 04:25:35.581568 8449 ts_tablet_manager.cc:1939] T 8a43ff2c91c9427db4edb21f1ad1f151 P cc00f1f932a34415b055f21185d7d3f1: Deleting consensus metadata
I20260620 04:25:35.581923 7901 catalog_manager.cc:4841] TS f76fa9ea200d4526b6d05f551c0478f9 (127.0.98.2:45413): tablet 8a43ff2c91c9427db4edb21f1ad1f151 (table test-table [id=8c82a51342fc48f4b2e8093e7c1f0fd6]) successfully deleted
I20260620 04:25:35.582633 7901 catalog_manager.cc:4841] TS cc00f1f932a34415b055f21185d7d3f1 (127.0.98.1:43319): tablet 8a43ff2c91c9427db4edb21f1ad1f151 (table test-table [id=8c82a51342fc48f4b2e8093e7c1f0fd6]) successfully deleted
I20260620 04:25:35.583104 8425 tablet_bootstrap.cc:492] T 1263a5d7a46842dbaeb7e3f78a40c613 P cc00f1f932a34415b055f21185d7d3f1: No bootstrap required, opened a new log
I20260620 04:25:35.583190 8425 ts_tablet_manager.cc:1397] T 1263a5d7a46842dbaeb7e3f78a40c613 P cc00f1f932a34415b055f21185d7d3f1: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260620 04:25:35.583645 8425 raft_consensus.cc:348] T 1263a5d7a46842dbaeb7e3f78a40c613 P cc00f1f932a34415b055f21185d7d3f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc00f1f932a34415b055f21185d7d3f1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43319 } } peers { permanent_uuid: "f76fa9ea200d4526b6d05f551c0478f9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45413 } } peers { permanent_uuid: "b6b68ffb3c80489f9a183bfcd20cafca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42135 } }
I20260620 04:25:35.583768 8425 raft_consensus.cc:374] T 1263a5d7a46842dbaeb7e3f78a40c613 P cc00f1f932a34415b055f21185d7d3f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:25:35.583806 8425 raft_consensus.cc:729] T 1263a5d7a46842dbaeb7e3f78a40c613 P cc00f1f932a34415b055f21185d7d3f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc00f1f932a34415b055f21185d7d3f1, State: Initialized, Role: FOLLOWER
I20260620 04:25:35.583915 8425 consensus_queue.cc:260] T 1263a5d7a46842dbaeb7e3f78a40c613 P cc00f1f932a34415b055f21185d7d3f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc00f1f932a34415b055f21185d7d3f1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43319 } } peers { permanent_uuid: "f76fa9ea200d4526b6d05f551c0478f9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45413 } } peers { permanent_uuid: "b6b68ffb3c80489f9a183bfcd20cafca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42135 } }
I20260620 04:25:35.584180 8425 ts_tablet_manager.cc:1428] T 1263a5d7a46842dbaeb7e3f78a40c613 P cc00f1f932a34415b055f21185d7d3f1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 04:25:35.584749 8427 tablet_bootstrap.cc:492] T 1263a5d7a46842dbaeb7e3f78a40c613 P f76fa9ea200d4526b6d05f551c0478f9: No bootstrap required, opened a new log
I20260620 04:25:35.584833 8427 ts_tablet_manager.cc:1397] T 1263a5d7a46842dbaeb7e3f78a40c613 P f76fa9ea200d4526b6d05f551c0478f9: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.001s
I20260620 04:25:35.585294 8427 raft_consensus.cc:348] T 1263a5d7a46842dbaeb7e3f78a40c613 P f76fa9ea200d4526b6d05f551c0478f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc00f1f932a34415b055f21185d7d3f1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43319 } } peers { permanent_uuid: "f76fa9ea200d4526b6d05f551c0478f9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45413 } } peers { permanent_uuid: "b6b68ffb3c80489f9a183bfcd20cafca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42135 } }
I20260620 04:25:35.585421 8427 raft_consensus.cc:374] T 1263a5d7a46842dbaeb7e3f78a40c613 P f76fa9ea200d4526b6d05f551c0478f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:25:35.585464 8427 raft_consensus.cc:729] T 1263a5d7a46842dbaeb7e3f78a40c613 P f76fa9ea200d4526b6d05f551c0478f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f76fa9ea200d4526b6d05f551c0478f9, State: Initialized, Role: FOLLOWER
I20260620 04:25:35.585582 8427 consensus_queue.cc:260] T 1263a5d7a46842dbaeb7e3f78a40c613 P f76fa9ea200d4526b6d05f551c0478f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc00f1f932a34415b055f21185d7d3f1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43319 } } peers { permanent_uuid: "f76fa9ea200d4526b6d05f551c0478f9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45413 } } peers { permanent_uuid: "b6b68ffb3c80489f9a183bfcd20cafca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42135 } }
I20260620 04:25:35.585839 8427 ts_tablet_manager.cc:1428] T 1263a5d7a46842dbaeb7e3f78a40c613 P f76fa9ea200d4526b6d05f551c0478f9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 04:25:35.687397 8431 raft_consensus.cc:482] T 1263a5d7a46842dbaeb7e3f78a40c613 P cc00f1f932a34415b055f21185d7d3f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:25:35.687544 8431 raft_consensus.cc:504] T 1263a5d7a46842dbaeb7e3f78a40c613 P cc00f1f932a34415b055f21185d7d3f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc00f1f932a34415b055f21185d7d3f1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43319 } } peers { permanent_uuid: "f76fa9ea200d4526b6d05f551c0478f9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45413 } } peers { permanent_uuid: "b6b68ffb3c80489f9a183bfcd20cafca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42135 } }
I20260620 04:25:35.688017 8431 leader_election.cc:288] T 1263a5d7a46842dbaeb7e3f78a40c613 P cc00f1f932a34415b055f21185d7d3f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f76fa9ea200d4526b6d05f551c0478f9 (127.0.98.2:45413), b6b68ffb3c80489f9a183bfcd20cafca (127.0.98.3:42135)
I20260620 04:25:35.688553 8211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1263a5d7a46842dbaeb7e3f78a40c613" candidate_uuid: "cc00f1f932a34415b055f21185d7d3f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f76fa9ea200d4526b6d05f551c0478f9" is_pre_election: true
I20260620 04:25:35.688566 8350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1263a5d7a46842dbaeb7e3f78a40c613" candidate_uuid: "cc00f1f932a34415b055f21185d7d3f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6b68ffb3c80489f9a183bfcd20cafca" is_pre_election: true
I20260620 04:25:35.688711 8211 raft_consensus.cc:2454] T 1263a5d7a46842dbaeb7e3f78a40c613 P f76fa9ea200d4526b6d05f551c0478f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc00f1f932a34415b055f21185d7d3f1 in term 0.
I20260620 04:25:35.688711 8350 raft_consensus.cc:2454] T 1263a5d7a46842dbaeb7e3f78a40c613 P b6b68ffb3c80489f9a183bfcd20cafca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc00f1f932a34415b055f21185d7d3f1 in term 0.
I20260620 04:25:35.689074 8005 leader_election.cc:302] T 1263a5d7a46842dbaeb7e3f78a40c613 P cc00f1f932a34415b055f21185d7d3f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cc00f1f932a34415b055f21185d7d3f1, f76fa9ea200d4526b6d05f551c0478f9; no voters:
I20260620 04:25:35.689267 8431 raft_consensus.cc:2781] T 1263a5d7a46842dbaeb7e3f78a40c613 P cc00f1f932a34415b055f21185d7d3f1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 04:25:35.689330 8431 raft_consensus.cc:482] T 1263a5d7a46842dbaeb7e3f78a40c613 P cc00f1f932a34415b055f21185d7d3f1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 04:25:35.689376 8431 raft_consensus.cc:3037] T 1263a5d7a46842dbaeb7e3f78a40c613 P cc00f1f932a34415b055f21185d7d3f1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:25:35.690452 8431 raft_consensus.cc:504] T 1263a5d7a46842dbaeb7e3f78a40c613 P cc00f1f932a34415b055f21185d7d3f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc00f1f932a34415b055f21185d7d3f1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43319 } } peers { permanent_uuid: "f76fa9ea200d4526b6d05f551c0478f9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45413 } } peers { permanent_uuid: "b6b68ffb3c80489f9a183bfcd20cafca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42135 } }
I20260620 04:25:35.690757 8431 leader_election.cc:288] T 1263a5d7a46842dbaeb7e3f78a40c613 P cc00f1f932a34415b055f21185d7d3f1 [CANDIDATE]: Term 1 election: Requested vote from peers f76fa9ea200d4526b6d05f551c0478f9 (127.0.98.2:45413), b6b68ffb3c80489f9a183bfcd20cafca (127.0.98.3:42135)
I20260620 04:25:35.691094 8350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1263a5d7a46842dbaeb7e3f78a40c613" candidate_uuid: "cc00f1f932a34415b055f21185d7d3f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6b68ffb3c80489f9a183bfcd20cafca"
I20260620 04:25:35.691094 8211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1263a5d7a46842dbaeb7e3f78a40c613" candidate_uuid: "cc00f1f932a34415b055f21185d7d3f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f76fa9ea200d4526b6d05f551c0478f9"
I20260620 04:25:35.691206 8350 raft_consensus.cc:3037] T 1263a5d7a46842dbaeb7e3f78a40c613 P b6b68ffb3c80489f9a183bfcd20cafca [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:25:35.691206 8211 raft_consensus.cc:3037] T 1263a5d7a46842dbaeb7e3f78a40c613 P f76fa9ea200d4526b6d05f551c0478f9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:25:35.692291 8350 raft_consensus.cc:2454] T 1263a5d7a46842dbaeb7e3f78a40c613 P b6b68ffb3c80489f9a183bfcd20cafca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc00f1f932a34415b055f21185d7d3f1 in term 1.
I20260620 04:25:35.692291 8211 raft_consensus.cc:2454] T 1263a5d7a46842dbaeb7e3f78a40c613 P f76fa9ea200d4526b6d05f551c0478f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc00f1f932a34415b055f21185d7d3f1 in term 1.
I20260620 04:25:35.692567 8004 leader_election.cc:302] T 1263a5d7a46842dbaeb7e3f78a40c613 P cc00f1f932a34415b055f21185d7d3f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b6b68ffb3c80489f9a183bfcd20cafca, cc00f1f932a34415b055f21185d7d3f1; no voters:
I20260620 04:25:35.692791 8431 raft_consensus.cc:2781] T 1263a5d7a46842dbaeb7e3f78a40c613 P cc00f1f932a34415b055f21185d7d3f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:25:35.692869 8431 raft_consensus.cc:686] T 1263a5d7a46842dbaeb7e3f78a40c613 P cc00f1f932a34415b055f21185d7d3f1 [term 1 LEADER]: Becoming Leader. State: Replica: cc00f1f932a34415b055f21185d7d3f1, State: Running, Role: LEADER
I20260620 04:25:35.692992 8431 consensus_queue.cc:237] T 1263a5d7a46842dbaeb7e3f78a40c613 P cc00f1f932a34415b055f21185d7d3f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc00f1f932a34415b055f21185d7d3f1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43319 } } peers { permanent_uuid: "f76fa9ea200d4526b6d05f551c0478f9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45413 } } peers { permanent_uuid: "b6b68ffb3c80489f9a183bfcd20cafca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42135 } }
I20260620 04:25:35.694686 7913 catalog_manager.cc:5495] T 1263a5d7a46842dbaeb7e3f78a40c613 P cc00f1f932a34415b055f21185d7d3f1 reported cstate change: term changed from 0 to 1, leader changed from <none> to cc00f1f932a34415b055f21185d7d3f1 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "cc00f1f932a34415b055f21185d7d3f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc00f1f932a34415b055f21185d7d3f1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f76fa9ea200d4526b6d05f551c0478f9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45413 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6b68ffb3c80489f9a183bfcd20cafca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42135 } health_report { overall_health: UNKNOWN } } }
Jun 20 04:25:35 dist-test-slave-nk6d krb5kdc[7869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929534, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 04:25:35.761535 8461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:35.749523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38521 (local address 127.0.0.1:35848)
0620 04:25:35.749798 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:35.749812 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:35.749919 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:35.750191 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:35.750193 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:35.750561 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:35.750655 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:35.750661 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:35.751443 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:35.751446 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:35.752076 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:35.752085 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:35.752509 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:35.752512 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:35.752587 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:35.757099 (+ 4512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:35.757112 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:35.758604 (+ 1492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:25:35.760164 (+ 1560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:35.760172 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:35.760177 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:35.760400 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:35.760623 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:35.760625 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:35.760626 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:35.760697 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:35.761029 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:25:35.761034 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:25:35.761144 (+ 110us) client_negotiation.cc:769] Sending connection context
0620 04:25:35.761246 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 04:25:35.761371 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":72,"threads_started":1}
I20260620 04:25:35.761560 8460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:35.749549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35848 (local address 127.0.98.62:38521)
0620 04:25:35.749744 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:35.749747 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:35.749873 (+ 126us) server_negotiation.cc:407] Connection header received
0620 04:25:35.750027 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:35.750029 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:35.750066 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:35.750165 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:35.750804 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:35.751332 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:35.752172 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:35.752394 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:35.752472 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:35.758752 (+ 6280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:35.758755 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:35.758757 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:25:35.758777 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:35.760061 (+ 1284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:35.760489 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:35.760491 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:35.760493 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:35.760560 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:35.760754 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:35.760756 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:35.760757 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:35.760921 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:35.760989 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:35.761286 (+ 297us) server_negotiation.cc:299] Negotiation successful
0620 04:25:35.761427 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":126,"thread_start_us":61,"threads_started":1}
I20260620 04:25:35.765152 7913 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35848:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 04:25:35.765832 7913 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 04:25:35.770309 8051 tablet_service.cc:1467] Processing CreateTablet for tablet 0e309de1d4e544999158a2fb02cf41d4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7262eb3c81fc419a871b24e8caa43e75]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 04:25:35.770393 8191 tablet_service.cc:1467] Processing CreateTablet for tablet 0e309de1d4e544999158a2fb02cf41d4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7262eb3c81fc419a871b24e8caa43e75]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 04:25:35.770393 8331 tablet_service.cc:1467] Processing CreateTablet for tablet 0e309de1d4e544999158a2fb02cf41d4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7262eb3c81fc419a871b24e8caa43e75]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 04:25:35.770690 8051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e309de1d4e544999158a2fb02cf41d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:25:35.770740 8191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e309de1d4e544999158a2fb02cf41d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:25:35.770740 8331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e309de1d4e544999158a2fb02cf41d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:25:35.773026 8425 tablet_bootstrap.cc:492] T 0e309de1d4e544999158a2fb02cf41d4 P cc00f1f932a34415b055f21185d7d3f1: Bootstrap starting.
I20260620 04:25:35.773437 8426 tablet_bootstrap.cc:492] T 0e309de1d4e544999158a2fb02cf41d4 P b6b68ffb3c80489f9a183bfcd20cafca: Bootstrap starting.
I20260620 04:25:35.773838 8425 tablet_bootstrap.cc:654] T 0e309de1d4e544999158a2fb02cf41d4 P cc00f1f932a34415b055f21185d7d3f1: Neither blocks nor log segments found. Creating new log.
I20260620 04:25:35.774307 8426 tablet_bootstrap.cc:654] T 0e309de1d4e544999158a2fb02cf41d4 P b6b68ffb3c80489f9a183bfcd20cafca: Neither blocks nor log segments found. Creating new log.
I20260620 04:25:35.774842 8427 tablet_bootstrap.cc:492] T 0e309de1d4e544999158a2fb02cf41d4 P f76fa9ea200d4526b6d05f551c0478f9: Bootstrap starting.
I20260620 04:25:35.774942 8425 tablet_bootstrap.cc:492] T 0e309de1d4e544999158a2fb02cf41d4 P cc00f1f932a34415b055f21185d7d3f1: No bootstrap required, opened a new log
I20260620 04:25:35.775020 8425 ts_tablet_manager.cc:1397] T 0e309de1d4e544999158a2fb02cf41d4 P cc00f1f932a34415b055f21185d7d3f1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 04:25:35.775424 8425 raft_consensus.cc:348] T 0e309de1d4e544999158a2fb02cf41d4 P cc00f1f932a34415b055f21185d7d3f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc00f1f932a34415b055f21185d7d3f1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43319 } } peers { permanent_uuid: "f76fa9ea200d4526b6d05f551c0478f9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45413 } } peers { permanent_uuid: "b6b68ffb3c80489f9a183bfcd20cafca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42135 } }
I20260620 04:25:35.775540 8425 raft_consensus.cc:374] T 0e309de1d4e544999158a2fb02cf41d4 P cc00f1f932a34415b055f21185d7d3f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:25:35.775581 8425 raft_consensus.cc:729] T 0e309de1d4e544999158a2fb02cf41d4 P cc00f1f932a34415b055f21185d7d3f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc00f1f932a34415b055f21185d7d3f1, State: Initialized, Role: FOLLOWER
I20260620 04:25:35.775733 8427 tablet_bootstrap.cc:654] T 0e309de1d4e544999158a2fb02cf41d4 P f76fa9ea200d4526b6d05f551c0478f9: Neither blocks nor log segments found. Creating new log.
I20260620 04:25:35.775709 8425 consensus_queue.cc:260] T 0e309de1d4e544999158a2fb02cf41d4 P cc00f1f932a34415b055f21185d7d3f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc00f1f932a34415b055f21185d7d3f1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43319 } } peers { permanent_uuid: "f76fa9ea200d4526b6d05f551c0478f9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45413 } } peers { permanent_uuid: "b6b68ffb3c80489f9a183bfcd20cafca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42135 } }
I20260620 04:25:35.776010 8425 ts_tablet_manager.cc:1428] T 0e309de1d4e544999158a2fb02cf41d4 P cc00f1f932a34415b055f21185d7d3f1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 04:25:35.775936 8431 tablet_replica.cc:439] T 0e309de1d4e544999158a2fb02cf41d4 P cc00f1f932a34415b055f21185d7d3f1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc00f1f932a34415b055f21185d7d3f1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43319 } } peers { permanent_uuid: "f76fa9ea200d4526b6d05f551c0478f9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45413 } } peers { permanent_uuid: "b6b68ffb3c80489f9a183bfcd20cafca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42135 } } }
I20260620 04:25:35.776117 8431 tablet_replica.cc:442] T 0e309de1d4e544999158a2fb02cf41d4 P cc00f1f932a34415b055f21185d7d3f1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 04:25:35.776975 8426 tablet_bootstrap.cc:492] T 0e309de1d4e544999158a2fb02cf41d4 P b6b68ffb3c80489f9a183bfcd20cafca: No bootstrap required, opened a new log
I20260620 04:25:35.777067 8426 ts_tablet_manager.cc:1397] T 0e309de1d4e544999158a2fb02cf41d4 P b6b68ffb3c80489f9a183bfcd20cafca: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260620 04:25:35.777156 8427 tablet_bootstrap.cc:492] T 0e309de1d4e544999158a2fb02cf41d4 P f76fa9ea200d4526b6d05f551c0478f9: No bootstrap required, opened a new log
I20260620 04:25:35.777240 8427 ts_tablet_manager.cc:1397] T 0e309de1d4e544999158a2fb02cf41d4 P f76fa9ea200d4526b6d05f551c0478f9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 04:25:35.777494 8426 raft_consensus.cc:348] T 0e309de1d4e544999158a2fb02cf41d4 P b6b68ffb3c80489f9a183bfcd20cafca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc00f1f932a34415b055f21185d7d3f1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43319 } } peers { permanent_uuid: "f76fa9ea200d4526b6d05f551c0478f9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45413 } } peers { permanent_uuid: "b6b68ffb3c80489f9a183bfcd20cafca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42135 } }
I20260620 04:25:35.777581 8426 raft_consensus.cc:374] T 0e309de1d4e544999158a2fb02cf41d4 P b6b68ffb3c80489f9a183bfcd20cafca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:25:35.777601 8426 raft_consensus.cc:729] T 0e309de1d4e544999158a2fb02cf41d4 P b6b68ffb3c80489f9a183bfcd20cafca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6b68ffb3c80489f9a183bfcd20cafca, State: Initialized, Role: FOLLOWER
I20260620 04:25:35.777630 8427 raft_consensus.cc:348] T 0e309de1d4e544999158a2fb02cf41d4 P f76fa9ea200d4526b6d05f551c0478f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc00f1f932a34415b055f21185d7d3f1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43319 } } peers { permanent_uuid: "f76fa9ea200d4526b6d05f551c0478f9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45413 } } peers { permanent_uuid: "b6b68ffb3c80489f9a183bfcd20cafca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42135 } }
I20260620 04:25:35.777737 8427 raft_consensus.cc:374] T 0e309de1d4e544999158a2fb02cf41d4 P f76fa9ea200d4526b6d05f551c0478f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:25:35.777720 8426 consensus_queue.cc:260] T 0e309de1d4e544999158a2fb02cf41d4 P b6b68ffb3c80489f9a183bfcd20cafca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc00f1f932a34415b055f21185d7d3f1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43319 } } peers { permanent_uuid: "f76fa9ea200d4526b6d05f551c0478f9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45413 } } peers { permanent_uuid: "b6b68ffb3c80489f9a183bfcd20cafca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42135 } }
I20260620 04:25:35.777776 8427 raft_consensus.cc:729] T 0e309de1d4e544999158a2fb02cf41d4 P f76fa9ea200d4526b6d05f551c0478f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f76fa9ea200d4526b6d05f551c0478f9, State: Initialized, Role: FOLLOWER
I20260620 04:25:35.777885 8427 consensus_queue.cc:260] T 0e309de1d4e544999158a2fb02cf41d4 P f76fa9ea200d4526b6d05f551c0478f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc00f1f932a34415b055f21185d7d3f1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43319 } } peers { permanent_uuid: "f76fa9ea200d4526b6d05f551c0478f9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45413 } } peers { permanent_uuid: "b6b68ffb3c80489f9a183bfcd20cafca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42135 } }
I20260620 04:25:35.777966 8426 ts_tablet_manager.cc:1428] T 0e309de1d4e544999158a2fb02cf41d4 P b6b68ffb3c80489f9a183bfcd20cafca: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 04:25:35.777933 8433 tablet_replica.cc:439] T 0e309de1d4e544999158a2fb02cf41d4 P b6b68ffb3c80489f9a183bfcd20cafca: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc00f1f932a34415b055f21185d7d3f1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43319 } } peers { permanent_uuid: "f76fa9ea200d4526b6d05f551c0478f9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45413 } } peers { permanent_uuid: "b6b68ffb3c80489f9a183bfcd20cafca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42135 } } }
I20260620 04:25:35.778075 8433 tablet_replica.cc:442] T 0e309de1d4e544999158a2fb02cf41d4 P b6b68ffb3c80489f9a183bfcd20cafca: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 04:25:35.778136 8427 ts_tablet_manager.cc:1428] T 0e309de1d4e544999158a2fb02cf41d4 P f76fa9ea200d4526b6d05f551c0478f9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 04:25:35.778306 8432 tablet_replica.cc:439] T 0e309de1d4e544999158a2fb02cf41d4 P f76fa9ea200d4526b6d05f551c0478f9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc00f1f932a34415b055f21185d7d3f1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43319 } } peers { permanent_uuid: "f76fa9ea200d4526b6d05f551c0478f9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45413 } } peers { permanent_uuid: "b6b68ffb3c80489f9a183bfcd20cafca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42135 } } }
I20260620 04:25:35.778412 8432 tablet_replica.cc:442] T 0e309de1d4e544999158a2fb02cf41d4 P f76fa9ea200d4526b6d05f551c0478f9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 04:25:35.874259 8433 raft_consensus.cc:482] T 0e309de1d4e544999158a2fb02cf41d4 P b6b68ffb3c80489f9a183bfcd20cafca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:25:35.874389 8433 raft_consensus.cc:504] T 0e309de1d4e544999158a2fb02cf41d4 P b6b68ffb3c80489f9a183bfcd20cafca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc00f1f932a34415b055f21185d7d3f1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43319 } } peers { permanent_uuid: "f76fa9ea200d4526b6d05f551c0478f9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45413 } } peers { permanent_uuid: "b6b68ffb3c80489f9a183bfcd20cafca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42135 } }
I20260620 04:25:35.875630 8433 leader_election.cc:288] T 0e309de1d4e544999158a2fb02cf41d4 P b6b68ffb3c80489f9a183bfcd20cafca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc00f1f932a34415b055f21185d7d3f1 (127.0.98.1:43319), f76fa9ea200d4526b6d05f551c0478f9 (127.0.98.2:45413)
I20260620 04:25:35.879243 8423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:35.875933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39807 (local address 127.0.98.1:43319)
0620 04:25:35.876097 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:35.876102 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:35.876252 (+ 150us) server_negotiation.cc:407] Connection header received
0620 04:25:35.876393 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:35.876396 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:35.876438 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:35.876501 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:25:35.876739 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:35.877242 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:35.878350 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:35.878758 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:35.878847 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:35.878912 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:35.879063 (+ 151us) server_negotiation.cc:299] Negotiation successful
0620 04:25:35.879130 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 04:25:35.879271 8465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:35.875887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43319 (local address 127.0.98.3:39807)
0620 04:25:35.876155 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:35.876187 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:35.876293 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:35.876518 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:35.876521 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:35.876537 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:25:35.876647 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:35.876652 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:35.877339 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:35.877343 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:35.878236 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:35.878245 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:35.878858 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:35.878862 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:35.878951 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:35.878975 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 04:25:35.879041 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 04:25:35.879108 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"mutex_wait_us":31,"thread_start_us":110,"threads_started":1}
I20260620 04:25:35.879484 8424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:35.876607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35811 (local address 127.0.98.2:45413)
0620 04:25:35.876721 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:35.876726 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:35.876738 (+ 12us) server_negotiation.cc:407] Connection header received
0620 04:25:35.876779 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:35.876780 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:35.876815 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:35.876874 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:25:35.877117 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:35.877631 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:35.878638 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:35.879016 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:35.879100 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:35.879156 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:35.879310 (+ 154us) server_negotiation.cc:299] Negotiation successful
0620 04:25:35.879372 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 04:25:35.879503 8466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:35.876264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45413 (local address 127.0.98.3:35811)
0620 04:25:35.876506 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:35.876520 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:35.876607 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:35.876891 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:35.876894 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:35.876908 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:25:35.877013 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:35.877019 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:35.877725 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:35.877729 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:35.878523 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:35.878531 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:35.879113 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:35.879117 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:35.879204 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:35.879224 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 04:25:35.879286 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 04:25:35.879348 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"mutex_wait_us":4,"thread_start_us":61,"threads_started":1}
I20260620 04:25:35.880075 8211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e309de1d4e544999158a2fb02cf41d4" candidate_uuid: "b6b68ffb3c80489f9a183bfcd20cafca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f76fa9ea200d4526b6d05f551c0478f9" is_pre_election: true
I20260620 04:25:35.880123 8071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e309de1d4e544999158a2fb02cf41d4" candidate_uuid: "b6b68ffb3c80489f9a183bfcd20cafca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc00f1f932a34415b055f21185d7d3f1" is_pre_election: true
I20260620 04:25:35.880229 8211 raft_consensus.cc:2454] T 0e309de1d4e544999158a2fb02cf41d4 P f76fa9ea200d4526b6d05f551c0478f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6b68ffb3c80489f9a183bfcd20cafca in term 0.
I20260620 04:25:35.880336 8071 raft_consensus.cc:2454] T 0e309de1d4e544999158a2fb02cf41d4 P cc00f1f932a34415b055f21185d7d3f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6b68ffb3c80489f9a183bfcd20cafca in term 0.
I20260620 04:25:35.880558 8285 leader_election.cc:302] T 0e309de1d4e544999158a2fb02cf41d4 P b6b68ffb3c80489f9a183bfcd20cafca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b6b68ffb3c80489f9a183bfcd20cafca, f76fa9ea200d4526b6d05f551c0478f9; no voters:
I20260620 04:25:35.880800 8433 raft_consensus.cc:2781] T 0e309de1d4e544999158a2fb02cf41d4 P b6b68ffb3c80489f9a183bfcd20cafca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 04:25:35.880854 8433 raft_consensus.cc:482] T 0e309de1d4e544999158a2fb02cf41d4 P b6b68ffb3c80489f9a183bfcd20cafca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 04:25:35.880879 8433 raft_consensus.cc:3037] T 0e309de1d4e544999158a2fb02cf41d4 P b6b68ffb3c80489f9a183bfcd20cafca [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:25:35.881837 8433 raft_consensus.cc:504] T 0e309de1d4e544999158a2fb02cf41d4 P b6b68ffb3c80489f9a183bfcd20cafca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc00f1f932a34415b055f21185d7d3f1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43319 } } peers { permanent_uuid: "f76fa9ea200d4526b6d05f551c0478f9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45413 } } peers { permanent_uuid: "b6b68ffb3c80489f9a183bfcd20cafca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42135 } }
I20260620 04:25:35.882143 8433 leader_election.cc:288] T 0e309de1d4e544999158a2fb02cf41d4 P b6b68ffb3c80489f9a183bfcd20cafca [CANDIDATE]: Term 1 election: Requested vote from peers cc00f1f932a34415b055f21185d7d3f1 (127.0.98.1:43319), f76fa9ea200d4526b6d05f551c0478f9 (127.0.98.2:45413)
I20260620 04:25:35.882501 8071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e309de1d4e544999158a2fb02cf41d4" candidate_uuid: "b6b68ffb3c80489f9a183bfcd20cafca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc00f1f932a34415b055f21185d7d3f1"
I20260620 04:25:35.882527 8211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e309de1d4e544999158a2fb02cf41d4" candidate_uuid: "b6b68ffb3c80489f9a183bfcd20cafca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f76fa9ea200d4526b6d05f551c0478f9"
I20260620 04:25:35.882663 8211 raft_consensus.cc:3037] T 0e309de1d4e544999158a2fb02cf41d4 P f76fa9ea200d4526b6d05f551c0478f9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:25:35.882663 8071 raft_consensus.cc:3037] T 0e309de1d4e544999158a2fb02cf41d4 P cc00f1f932a34415b055f21185d7d3f1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:25:35.883724 8071 raft_consensus.cc:2454] T 0e309de1d4e544999158a2fb02cf41d4 P cc00f1f932a34415b055f21185d7d3f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6b68ffb3c80489f9a183bfcd20cafca in term 1.
I20260620 04:25:35.883724 8211 raft_consensus.cc:2454] T 0e309de1d4e544999158a2fb02cf41d4 P f76fa9ea200d4526b6d05f551c0478f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6b68ffb3c80489f9a183bfcd20cafca in term 1.
I20260620 04:25:35.884002 8285 leader_election.cc:302] T 0e309de1d4e544999158a2fb02cf41d4 P b6b68ffb3c80489f9a183bfcd20cafca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b6b68ffb3c80489f9a183bfcd20cafca, cc00f1f932a34415b055f21185d7d3f1; no voters:
I20260620 04:25:35.884160 8433 raft_consensus.cc:2781] T 0e309de1d4e544999158a2fb02cf41d4 P b6b68ffb3c80489f9a183bfcd20cafca [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:25:35.884375 8433 raft_consensus.cc:686] T 0e309de1d4e544999158a2fb02cf41d4 P b6b68ffb3c80489f9a183bfcd20cafca [term 1 LEADER]: Becoming Leader. State: Replica: b6b68ffb3c80489f9a183bfcd20cafca, State: Running, Role: LEADER
I20260620 04:25:35.884627 8433 consensus_queue.cc:237] T 0e309de1d4e544999158a2fb02cf41d4 P b6b68ffb3c80489f9a183bfcd20cafca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc00f1f932a34415b055f21185d7d3f1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43319 } } peers { permanent_uuid: "f76fa9ea200d4526b6d05f551c0478f9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45413 } } peers { permanent_uuid: "b6b68ffb3c80489f9a183bfcd20cafca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42135 } }
I20260620 04:25:35.885799 8467 tablet_replica.cc:439] T 0e309de1d4e544999158a2fb02cf41d4 P b6b68ffb3c80489f9a183bfcd20cafca: TxnStatusTablet state changed. Reason: New leader b6b68ffb3c80489f9a183bfcd20cafca. Latest consensus state: current_term: 1 leader_uuid: "b6b68ffb3c80489f9a183bfcd20cafca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc00f1f932a34415b055f21185d7d3f1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43319 } } peers { permanent_uuid: "f76fa9ea200d4526b6d05f551c0478f9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45413 } } peers { permanent_uuid: "b6b68ffb3c80489f9a183bfcd20cafca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42135 } } }
I20260620 04:25:35.885921 8467 tablet_replica.cc:442] T 0e309de1d4e544999158a2fb02cf41d4 P b6b68ffb3c80489f9a183bfcd20cafca: This TxnStatusTablet replica's current role is: LEADER
I20260620 04:25:35.886226 8469 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 04:25:35.886371 8469 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 04:25:35.886926 7913 catalog_manager.cc:5495] T 0e309de1d4e544999158a2fb02cf41d4 P b6b68ffb3c80489f9a183bfcd20cafca reported cstate change: term changed from 0 to 1, leader changed from <none> to b6b68ffb3c80489f9a183bfcd20cafca (127.0.98.3). New cstate: current_term: 1 leader_uuid: "b6b68ffb3c80489f9a183bfcd20cafca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc00f1f932a34415b055f21185d7d3f1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f76fa9ea200d4526b6d05f551c0478f9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45413 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6b68ffb3c80489f9a183bfcd20cafca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42135 } health_report { overall_health: HEALTHY } } }
I20260620 04:25:35.906649 8461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:35.903291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42135 (local address 127.0.0.1:60252)
0620 04:25:35.903450 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:35.903466 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:35.903557 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:35.903836 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:35.903838 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:35.903845 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 04:25:35.903933 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:35.903937 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:35.904627 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:35.904631 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:35.905240 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:35.905247 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:35.905640 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:35.905643 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:35.905725 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:35.905765 (+ 40us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 04:25:35.906369 (+ 604us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 04:25:35.906384 (+ 15us) client_negotiation.cc:769] Sending connection context
0620 04:25:35.906446 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 04:25:35.906528 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 04:25:35.906649 8421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:35.903398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60252 (local address 127.0.98.3:42135)
0620 04:25:35.903523 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:35.903527 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:35.903543 (+ 16us) server_negotiation.cc:407] Connection header received
0620 04:25:35.903671 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:35.903674 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:35.903732 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:35.903834 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:25:35.904035 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:35.904527 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:35.905345 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:35.905550 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:35.905641 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:35.905892 (+ 251us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:25:35.906284 (+ 392us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:25:35.906387 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:35.906455 (+ 68us) server_negotiation.cc:299] Negotiation successful
0620 04:25:35.906506 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 04:25:35.909194 8211 raft_consensus.cc:1261] T 0e309de1d4e544999158a2fb02cf41d4 P f76fa9ea200d4526b6d05f551c0478f9 [term 1 FOLLOWER]: Refusing update from remote peer b6b68ffb3c80489f9a183bfcd20cafca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:25:35.909277 8071 raft_consensus.cc:1261] T 0e309de1d4e544999158a2fb02cf41d4 P cc00f1f932a34415b055f21185d7d3f1 [term 1 FOLLOWER]: Refusing update from remote peer b6b68ffb3c80489f9a183bfcd20cafca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:25:35.909701 8433 consensus_queue.cc:1035] T 0e309de1d4e544999158a2fb02cf41d4 P b6b68ffb3c80489f9a183bfcd20cafca [LEADER]: Connected to new peer: Peer: permanent_uuid: "f76fa9ea200d4526b6d05f551c0478f9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:25:35.910050 8467 consensus_queue.cc:1035] T 0e309de1d4e544999158a2fb02cf41d4 P b6b68ffb3c80489f9a183bfcd20cafca [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc00f1f932a34415b055f21185d7d3f1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:25:35.911073 8432 tablet_replica.cc:439] T 0e309de1d4e544999158a2fb02cf41d4 P f76fa9ea200d4526b6d05f551c0478f9: TxnStatusTablet state changed. Reason: New leader b6b68ffb3c80489f9a183bfcd20cafca. Latest consensus state: current_term: 1 leader_uuid: "b6b68ffb3c80489f9a183bfcd20cafca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc00f1f932a34415b055f21185d7d3f1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43319 } } peers { permanent_uuid: "f76fa9ea200d4526b6d05f551c0478f9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45413 } } peers { permanent_uuid: "b6b68ffb3c80489f9a183bfcd20cafca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42135 } } }
I20260620 04:25:35.911191 8432 tablet_replica.cc:442] T 0e309de1d4e544999158a2fb02cf41d4 P f76fa9ea200d4526b6d05f551c0478f9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 04:25:35.911870 8431 tablet_replica.cc:439] T 0e309de1d4e544999158a2fb02cf41d4 P cc00f1f932a34415b055f21185d7d3f1: TxnStatusTablet state changed. Reason: New leader b6b68ffb3c80489f9a183bfcd20cafca. Latest consensus state: current_term: 1 leader_uuid: "b6b68ffb3c80489f9a183bfcd20cafca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc00f1f932a34415b055f21185d7d3f1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43319 } } peers { permanent_uuid: "f76fa9ea200d4526b6d05f551c0478f9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45413 } } peers { permanent_uuid: "b6b68ffb3c80489f9a183bfcd20cafca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42135 } } }
I20260620 04:25:35.912616 8433 tablet_replica.cc:439] T 0e309de1d4e544999158a2fb02cf41d4 P b6b68ffb3c80489f9a183bfcd20cafca: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b6b68ffb3c80489f9a183bfcd20cafca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc00f1f932a34415b055f21185d7d3f1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43319 } } peers { permanent_uuid: "f76fa9ea200d4526b6d05f551c0478f9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45413 } } peers { permanent_uuid: "b6b68ffb3c80489f9a183bfcd20cafca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42135 } } }
I20260620 04:25:35.912714 8433 tablet_replica.cc:442] T 0e309de1d4e544999158a2fb02cf41d4 P b6b68ffb3c80489f9a183bfcd20cafca: This TxnStatusTablet replica's current role is: LEADER
I20260620 04:25:35.913378 8431 tablet_replica.cc:442] T 0e309de1d4e544999158a2fb02cf41d4 P cc00f1f932a34415b055f21185d7d3f1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 04:25:35.915819 8440 mvcc.cc:205] Tried to move back new op lower bound from 7298783379079139328 to 7298783378985783296. Current Snapshot: MvccSnapshot[applied={T|T < 7298783379079139328 or (T in {7298783379079139328})}]
I20260620 04:25:35.916054 8432 tablet_replica.cc:439] T 0e309de1d4e544999158a2fb02cf41d4 P f76fa9ea200d4526b6d05f551c0478f9: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b6b68ffb3c80489f9a183bfcd20cafca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc00f1f932a34415b055f21185d7d3f1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43319 } } peers { permanent_uuid: "f76fa9ea200d4526b6d05f551c0478f9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45413 } } peers { permanent_uuid: "b6b68ffb3c80489f9a183bfcd20cafca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42135 } } }
I20260620 04:25:35.916162 8432 tablet_replica.cc:442] T 0e309de1d4e544999158a2fb02cf41d4 P f76fa9ea200d4526b6d05f551c0478f9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 04:25:35.918413 8467 tablet_replica.cc:439] T 0e309de1d4e544999158a2fb02cf41d4 P b6b68ffb3c80489f9a183bfcd20cafca: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b6b68ffb3c80489f9a183bfcd20cafca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc00f1f932a34415b055f21185d7d3f1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43319 } } peers { permanent_uuid: "f76fa9ea200d4526b6d05f551c0478f9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45413 } } peers { permanent_uuid: "b6b68ffb3c80489f9a183bfcd20cafca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42135 } } }
I20260620 04:25:35.918519 8467 tablet_replica.cc:442] T 0e309de1d4e544999158a2fb02cf41d4 P b6b68ffb3c80489f9a183bfcd20cafca: This TxnStatusTablet replica's current role is: LEADER
I20260620 04:25:35.919179 8431 tablet_replica.cc:439] T 0e309de1d4e544999158a2fb02cf41d4 P cc00f1f932a34415b055f21185d7d3f1: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b6b68ffb3c80489f9a183bfcd20cafca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc00f1f932a34415b055f21185d7d3f1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43319 } } peers { permanent_uuid: "f76fa9ea200d4526b6d05f551c0478f9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45413 } } peers { permanent_uuid: "b6b68ffb3c80489f9a183bfcd20cafca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42135 } } }
I20260620 04:25:35.919291 8431 tablet_replica.cc:442] T 0e309de1d4e544999158a2fb02cf41d4 P cc00f1f932a34415b055f21185d7d3f1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 04:25:35.926759 8421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:35.923772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57445 (local address 127.0.98.3:42135)
0620 04:25:35.923876 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:35.923880 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:35.924073 (+ 193us) server_negotiation.cc:407] Connection header received
0620 04:25:35.924212 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:35.924215 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:35.924257 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:35.924313 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:25:35.924524 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:35.925005 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:35.925834 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:35.926044 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:35.926132 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:35.926386 (+ 254us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:25:35.926451 (+ 65us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:25:35.926506 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:35.926589 (+ 83us) server_negotiation.cc:299] Negotiation successful
0620 04:25:35.926635 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 04:25:35.926786 8477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:35.923707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42135 (local address 127.0.98.1:57445)
0620 04:25:35.924000 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:35.924013 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:35.924111 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:35.924326 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:35.924329 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:35.924337 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 04:25:35.924437 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:35.924443 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:35.925098 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:35.925102 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:35.925730 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:35.925738 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:35.926133 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:35.926136 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:35.926235 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:35.926273 (+ 38us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 04:25:35.926519 (+ 246us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 04:25:35.926525 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 04:25:35.926580 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 04:25:35.926668 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":104,"threads_started":1}
I20260620 04:25:35.932845 8350 raft_consensus.cc:1261] T 1263a5d7a46842dbaeb7e3f78a40c613 P b6b68ffb3c80489f9a183bfcd20cafca [term 1 FOLLOWER]: Refusing update from remote peer cc00f1f932a34415b055f21185d7d3f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:25:35.933056 8210 raft_consensus.cc:1261] T 1263a5d7a46842dbaeb7e3f78a40c613 P f76fa9ea200d4526b6d05f551c0478f9 [term 1 FOLLOWER]: Refusing update from remote peer cc00f1f932a34415b055f21185d7d3f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:25:35.933233 8431 consensus_queue.cc:1035] T 1263a5d7a46842dbaeb7e3f78a40c613 P cc00f1f932a34415b055f21185d7d3f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6b68ffb3c80489f9a183bfcd20cafca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:25:35.933487 8431 consensus_queue.cc:1035] T 1263a5d7a46842dbaeb7e3f78a40c613 P cc00f1f932a34415b055f21185d7d3f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f76fa9ea200d4526b6d05f551c0478f9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:25:35.938061 8439 mvcc.cc:205] Tried to move back new op lower bound from 7298783379176366080 to 7298783378199502848. Current Snapshot: MvccSnapshot[applied={T|T < 7298783379176366080}]
I20260620 04:25:35.954399 8481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:35.950631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43319 (local address 127.0.98.3:56783)
0620 04:25:35.950988 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:35.951002 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:35.951105 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:35.951308 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:35.951311 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:35.951317 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 04:25:35.951440 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:35.951446 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:35.952166 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:35.952170 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:35.953180 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:35.953189 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:35.953697 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:35.953701 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:35.953822 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:35.953863 (+ 41us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 04:25:35.954121 (+ 258us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 04:25:35.954128 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 04:25:35.954184 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 04:25:35.954255 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":105,"threads_started":1}
I20260620 04:25:35.954380 8423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:35.950700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56783 (local address 127.0.98.1:43319)
0620 04:25:35.950806 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:35.950811 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:35.951064 (+ 253us) server_negotiation.cc:407] Connection header received
0620 04:25:35.951191 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:35.951194 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:35.951237 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:35.951287 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:25:35.951548 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:35.952084 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:35.953293 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:35.953609 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:35.953700 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:35.953982 (+ 282us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:25:35.954055 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:25:35.954109 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:35.954192 (+ 83us) server_negotiation.cc:299] Negotiation successful
0620 04:25:35.954243 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 04:25:35.973959 7914 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35844:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:25:35.974133 7914 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35844:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:25:35.976002 7914 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ac9ebe4a2398411f93639541ab97f9e7: Sending DeleteTablet for 3 replicas of tablet 1263a5d7a46842dbaeb7e3f78a40c613
I20260620 04:25:35.976456 8051 tablet_service.cc:1514] Processing DeleteTablet for tablet 1263a5d7a46842dbaeb7e3f78a40c613 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:25:35 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33054
I20260620 04:25:35.976532 8191 tablet_service.cc:1514] Processing DeleteTablet for tablet 1263a5d7a46842dbaeb7e3f78a40c613 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:25:35 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49960
I20260620 04:25:35.976629 8449 tablet_replica.cc:330] T 1263a5d7a46842dbaeb7e3f78a40c613 P cc00f1f932a34415b055f21185d7d3f1: stopping tablet replica
I20260620 04:25:35.976696 8450 tablet_replica.cc:330] T 1263a5d7a46842dbaeb7e3f78a40c613 P f76fa9ea200d4526b6d05f551c0478f9: stopping tablet replica
I20260620 04:25:35.976733 8449 raft_consensus.cc:2229] T 1263a5d7a46842dbaeb7e3f78a40c613 P cc00f1f932a34415b055f21185d7d3f1 [term 1 LEADER]: Raft consensus shutting down.
I20260620 04:25:35.976761 8331 tablet_service.cc:1514] Processing DeleteTablet for tablet 1263a5d7a46842dbaeb7e3f78a40c613 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:25:35 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60246
I20260620 04:25:35.976791 8450 raft_consensus.cc:2229] T 1263a5d7a46842dbaeb7e3f78a40c613 P f76fa9ea200d4526b6d05f551c0478f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:25:35.976884 8450 raft_consensus.cc:2258] T 1263a5d7a46842dbaeb7e3f78a40c613 P f76fa9ea200d4526b6d05f551c0478f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:25:35.976886 8449 raft_consensus.cc:2258] T 1263a5d7a46842dbaeb7e3f78a40c613 P cc00f1f932a34415b055f21185d7d3f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:25:35.976913 8448 tablet_replica.cc:330] T 1263a5d7a46842dbaeb7e3f78a40c613 P b6b68ffb3c80489f9a183bfcd20cafca: stopping tablet replica
I20260620 04:25:35.977013 8448 raft_consensus.cc:2229] T 1263a5d7a46842dbaeb7e3f78a40c613 P b6b68ffb3c80489f9a183bfcd20cafca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:25:35.977116 8448 raft_consensus.cc:2258] T 1263a5d7a46842dbaeb7e3f78a40c613 P b6b68ffb3c80489f9a183bfcd20cafca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:25:35.977698 8450 ts_tablet_manager.cc:1905] T 1263a5d7a46842dbaeb7e3f78a40c613 P f76fa9ea200d4526b6d05f551c0478f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:25:35.977717 8448 ts_tablet_manager.cc:1905] T 1263a5d7a46842dbaeb7e3f78a40c613 P b6b68ffb3c80489f9a183bfcd20cafca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:25:35.979858 8448 ts_tablet_manager.cc:1918] T 1263a5d7a46842dbaeb7e3f78a40c613 P b6b68ffb3c80489f9a183bfcd20cafca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 04:25:35.979921 8448 log.cc:1196] T 1263a5d7a46842dbaeb7e3f78a40c613 P b6b68ffb3c80489f9a183bfcd20cafca: Deleting WAL directory at /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929496330398-392-0/minicluster-data/ts-2/wal/wals/1263a5d7a46842dbaeb7e3f78a40c613
I20260620 04:25:35.980154 8448 ts_tablet_manager.cc:1939] T 1263a5d7a46842dbaeb7e3f78a40c613 P b6b68ffb3c80489f9a183bfcd20cafca: Deleting consensus metadata
I20260620 04:25:35.980388 8449 ts_tablet_manager.cc:1905] T 1263a5d7a46842dbaeb7e3f78a40c613 P cc00f1f932a34415b055f21185d7d3f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:25:35.980700 7900 catalog_manager.cc:4841] TS b6b68ffb3c80489f9a183bfcd20cafca (127.0.98.3:42135): tablet 1263a5d7a46842dbaeb7e3f78a40c613 (table test-table [id=b916be3080cf40a2a6d1f91be609e5d3]) successfully deleted
I20260620 04:25:35.982880 8449 ts_tablet_manager.cc:1918] T 1263a5d7a46842dbaeb7e3f78a40c613 P cc00f1f932a34415b055f21185d7d3f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 04:25:35.982954 8449 log.cc:1196] T 1263a5d7a46842dbaeb7e3f78a40c613 P cc00f1f932a34415b055f21185d7d3f1: Deleting WAL directory at /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929496330398-392-0/minicluster-data/ts-0/wal/wals/1263a5d7a46842dbaeb7e3f78a40c613
I20260620 04:25:35.983171 8449 ts_tablet_manager.cc:1939] T 1263a5d7a46842dbaeb7e3f78a40c613 P cc00f1f932a34415b055f21185d7d3f1: Deleting consensus metadata
I20260620 04:25:35.983814 7901 catalog_manager.cc:4841] TS cc00f1f932a34415b055f21185d7d3f1 (127.0.98.1:43319): tablet 1263a5d7a46842dbaeb7e3f78a40c613 (table test-table [id=b916be3080cf40a2a6d1f91be609e5d3]) successfully deleted
I20260620 04:25:35.984769 8450 ts_tablet_manager.cc:1918] T 1263a5d7a46842dbaeb7e3f78a40c613 P f76fa9ea200d4526b6d05f551c0478f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 04:25:35.984838 8450 log.cc:1196] T 1263a5d7a46842dbaeb7e3f78a40c613 P f76fa9ea200d4526b6d05f551c0478f9: Deleting WAL directory at /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929496330398-392-0/minicluster-data/ts-1/wal/wals/1263a5d7a46842dbaeb7e3f78a40c613
I20260620 04:25:35.985049 8450 ts_tablet_manager.cc:1939] T 1263a5d7a46842dbaeb7e3f78a40c613 P f76fa9ea200d4526b6d05f551c0478f9: Deleting consensus metadata
I20260620 04:25:35.985653 7901 catalog_manager.cc:4841] TS f76fa9ea200d4526b6d05f551c0478f9 (127.0.98.2:45413): tablet 1263a5d7a46842dbaeb7e3f78a40c613 (table test-table [id=b916be3080cf40a2a6d1f91be609e5d3]) successfully deleted
Jun 20 04:25:35 dist-test-slave-nk6d krb5kdc[7869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929535, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 04:25:36 dist-test-slave-nk6d krb5kdc[7869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:25:36 dist-test-slave-nk6d krb5kdc[7869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 04:25:36.011917 8460 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:25:36.005762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35850 (local address 127.0.98.62:38521)
0620 04:25:36.005884 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:36.005889 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:36.006063 (+ 174us) server_negotiation.cc:407] Connection header received
0620 04:25:36.006215 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:36.006218 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:36.006263 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:36.006327 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:36.007034 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:36.007558 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:36.008490 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:36.008726 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:36.008815 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:36.011796 (+ 2981us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35850: BlockingRecv error: recv got EOF from 127.0.0.1:35850 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Jun 20 04:25:36 dist-test-slave-nk6d krb5kdc[7869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929535, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 04:25:36.022411 8460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:36.013511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35862 (local address 127.0.98.62:38521)
0620 04:25:36.013613 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:36.013617 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:36.013977 (+ 360us) server_negotiation.cc:407] Connection header received
0620 04:25:36.014173 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:36.014176 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:36.014223 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:36.014306 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:36.014974 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:36.015483 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:36.016650 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:36.016855 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:36.016928 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:36.019485 (+ 2557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:36.019488 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:36.019490 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:25:36.019514 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:36.020892 (+ 1378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:36.021359 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:36.021361 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:36.021362 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:36.021431 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:36.021697 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:36.021699 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:36.021700 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:36.021854 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:36.021946 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:36.022190 (+ 244us) server_negotiation.cc:299] Negotiation successful
0620 04:25:36.022288 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 04:25:36.025179 7914 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35862:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 04:25:36.025600 7914 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 04:25:36.030285 8191 tablet_service.cc:1467] Processing CreateTablet for tablet ed9ddecf3b5f4b2aac970d1955a6a6fc (DEFAULT_TABLE table=test-table [id=6064e6a340bf4309ae980c790db4a755]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:25:36.030285 8331 tablet_service.cc:1467] Processing CreateTablet for tablet ed9ddecf3b5f4b2aac970d1955a6a6fc (DEFAULT_TABLE table=test-table [id=6064e6a340bf4309ae980c790db4a755]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:25:36.030336 8051 tablet_service.cc:1467] Processing CreateTablet for tablet ed9ddecf3b5f4b2aac970d1955a6a6fc (DEFAULT_TABLE table=test-table [id=6064e6a340bf4309ae980c790db4a755]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:25:36.030561 8191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed9ddecf3b5f4b2aac970d1955a6a6fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:25:36.030561 8331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed9ddecf3b5f4b2aac970d1955a6a6fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:25:36.030649 8051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed9ddecf3b5f4b2aac970d1955a6a6fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:25:36.032992 8427 tablet_bootstrap.cc:492] T ed9ddecf3b5f4b2aac970d1955a6a6fc P f76fa9ea200d4526b6d05f551c0478f9: Bootstrap starting.
I20260620 04:25:36.033202 8425 tablet_bootstrap.cc:492] T ed9ddecf3b5f4b2aac970d1955a6a6fc P cc00f1f932a34415b055f21185d7d3f1: Bootstrap starting.
I20260620 04:25:36.033521 8426 tablet_bootstrap.cc:492] T ed9ddecf3b5f4b2aac970d1955a6a6fc P b6b68ffb3c80489f9a183bfcd20cafca: Bootstrap starting.
I20260620 04:25:36.033859 8427 tablet_bootstrap.cc:654] T ed9ddecf3b5f4b2aac970d1955a6a6fc P f76fa9ea200d4526b6d05f551c0478f9: Neither blocks nor log segments found. Creating new log.
I20260620 04:25:36.034055 8425 tablet_bootstrap.cc:654] T ed9ddecf3b5f4b2aac970d1955a6a6fc P cc00f1f932a34415b055f21185d7d3f1: Neither blocks nor log segments found. Creating new log.
I20260620 04:25:36.034379 8426 tablet_bootstrap.cc:654] T ed9ddecf3b5f4b2aac970d1955a6a6fc P b6b68ffb3c80489f9a183bfcd20cafca: Neither blocks nor log segments found. Creating new log.
I20260620 04:25:36.035809 8426 tablet_bootstrap.cc:492] T ed9ddecf3b5f4b2aac970d1955a6a6fc P b6b68ffb3c80489f9a183bfcd20cafca: No bootstrap required, opened a new log
I20260620 04:25:36.035809 8427 tablet_bootstrap.cc:492] T ed9ddecf3b5f4b2aac970d1955a6a6fc P f76fa9ea200d4526b6d05f551c0478f9: No bootstrap required, opened a new log
I20260620 04:25:36.035887 8427 ts_tablet_manager.cc:1397] T ed9ddecf3b5f4b2aac970d1955a6a6fc P f76fa9ea200d4526b6d05f551c0478f9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 04:25:36.035887 8426 ts_tablet_manager.cc:1397] T ed9ddecf3b5f4b2aac970d1955a6a6fc P b6b68ffb3c80489f9a183bfcd20cafca: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 04:25:36.036340 8426 raft_consensus.cc:348] T ed9ddecf3b5f4b2aac970d1955a6a6fc P b6b68ffb3c80489f9a183bfcd20cafca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc00f1f932a34415b055f21185d7d3f1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43319 } } peers { permanent_uuid: "f76fa9ea200d4526b6d05f551c0478f9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45413 } } peers { permanent_uuid: "b6b68ffb3c80489f9a183bfcd20cafca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42135 } }
I20260620 04:25:36.036339 8427 raft_consensus.cc:348] T ed9ddecf3b5f4b2aac970d1955a6a6fc P f76fa9ea200d4526b6d05f551c0478f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc00f1f932a34415b055f21185d7d3f1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43319 } } peers { permanent_uuid: "f76fa9ea200d4526b6d05f551c0478f9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45413 } } peers { permanent_uuid: "b6b68ffb3c80489f9a183bfcd20cafca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42135 } }
I20260620 04:25:36.036412 8425 tablet_bootstrap.cc:492] T ed9ddecf3b5f4b2aac970d1955a6a6fc P cc00f1f932a34415b055f21185d7d3f1: No bootstrap required, opened a new log
I20260620 04:25:36.036441 8426 raft_consensus.cc:374] T ed9ddecf3b5f4b2aac970d1955a6a6fc P b6b68ffb3c80489f9a183bfcd20cafca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:25:36.036441 8427 raft_consensus.cc:374] T ed9ddecf3b5f4b2aac970d1955a6a6fc P f76fa9ea200d4526b6d05f551c0478f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:25:36.036473 8427 raft_consensus.cc:729] T ed9ddecf3b5f4b2aac970d1955a6a6fc P f76fa9ea200d4526b6d05f551c0478f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f76fa9ea200d4526b6d05f551c0478f9, State: Initialized, Role: FOLLOWER
I20260620 04:25:36.036492 8425 ts_tablet_manager.cc:1397] T ed9ddecf3b5f4b2aac970d1955a6a6fc P cc00f1f932a34415b055f21185d7d3f1: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 04:25:36.036473 8426 raft_consensus.cc:729] T ed9ddecf3b5f4b2aac970d1955a6a6fc P b6b68ffb3c80489f9a183bfcd20cafca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6b68ffb3c80489f9a183bfcd20cafca, State: Initialized, Role: FOLLOWER
I20260620 04:25:36.036592 8426 consensus_queue.cc:260] T ed9ddecf3b5f4b2aac970d1955a6a6fc P b6b68ffb3c80489f9a183bfcd20cafca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc00f1f932a34415b055f21185d7d3f1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43319 } } peers { permanent_uuid: "f76fa9ea200d4526b6d05f551c0478f9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45413 } } peers { permanent_uuid: "b6b68ffb3c80489f9a183bfcd20cafca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42135 } }
I20260620 04:25:36.036592 8427 consensus_queue.cc:260] T ed9ddecf3b5f4b2aac970d1955a6a6fc P f76fa9ea200d4526b6d05f551c0478f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc00f1f932a34415b055f21185d7d3f1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43319 } } peers { permanent_uuid: "f76fa9ea200d4526b6d05f551c0478f9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45413 } } peers { permanent_uuid: "b6b68ffb3c80489f9a183bfcd20cafca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42135 } }
I20260620 04:25:36.036837 8427 ts_tablet_manager.cc:1428] T ed9ddecf3b5f4b2aac970d1955a6a6fc P f76fa9ea200d4526b6d05f551c0478f9: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 04:25:36.036837 8426 ts_tablet_manager.cc:1428] T ed9ddecf3b5f4b2aac970d1955a6a6fc P b6b68ffb3c80489f9a183bfcd20cafca: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 04:25:36.036831 8425 raft_consensus.cc:348] T ed9ddecf3b5f4b2aac970d1955a6a6fc P cc00f1f932a34415b055f21185d7d3f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc00f1f932a34415b055f21185d7d3f1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43319 } } peers { permanent_uuid: "f76fa9ea200d4526b6d05f551c0478f9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45413 } } peers { permanent_uuid: "b6b68ffb3c80489f9a183bfcd20cafca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42135 } }
I20260620 04:25:36.036926 8425 raft_consensus.cc:374] T ed9ddecf3b5f4b2aac970d1955a6a6fc P cc00f1f932a34415b055f21185d7d3f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:25:36.036952 8425 raft_consensus.cc:729] T ed9ddecf3b5f4b2aac970d1955a6a6fc P cc00f1f932a34415b055f21185d7d3f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc00f1f932a34415b055f21185d7d3f1, State: Initialized, Role: FOLLOWER
I20260620 04:25:36.037051 8425 consensus_queue.cc:260] T ed9ddecf3b5f4b2aac970d1955a6a6fc P cc00f1f932a34415b055f21185d7d3f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc00f1f932a34415b055f21185d7d3f1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43319 } } peers { permanent_uuid: "f76fa9ea200d4526b6d05f551c0478f9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45413 } } peers { permanent_uuid: "b6b68ffb3c80489f9a183bfcd20cafca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42135 } }
I20260620 04:25:36.037272 8425 ts_tablet_manager.cc:1428] T ed9ddecf3b5f4b2aac970d1955a6a6fc P cc00f1f932a34415b055f21185d7d3f1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 04:25:36.372146 8432 raft_consensus.cc:482] T ed9ddecf3b5f4b2aac970d1955a6a6fc P f76fa9ea200d4526b6d05f551c0478f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:25:36.372310 8432 raft_consensus.cc:504] T ed9ddecf3b5f4b2aac970d1955a6a6fc P f76fa9ea200d4526b6d05f551c0478f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc00f1f932a34415b055f21185d7d3f1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43319 } } peers { permanent_uuid: "f76fa9ea200d4526b6d05f551c0478f9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45413 } } peers { permanent_uuid: "b6b68ffb3c80489f9a183bfcd20cafca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42135 } }
I20260620 04:25:36.373407 8432 leader_election.cc:288] T ed9ddecf3b5f4b2aac970d1955a6a6fc P f76fa9ea200d4526b6d05f551c0478f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc00f1f932a34415b055f21185d7d3f1 (127.0.98.1:43319), b6b68ffb3c80489f9a183bfcd20cafca (127.0.98.3:42135)
I20260620 04:25:36.378579 8423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:36.373860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46855 (local address 127.0.98.1:43319)
0620 04:25:36.373962 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:36.373965 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:36.374281 (+ 316us) server_negotiation.cc:407] Connection header received
0620 04:25:36.374373 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:36.374376 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:36.374423 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:36.374490 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:25:36.374808 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:36.375577 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:36.377336 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:36.377940 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:36.378047 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:36.378126 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:36.378373 (+ 247us) server_negotiation.cc:299] Negotiation successful
0620 04:25:36.378471 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 04:25:36.378579 8421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:36.373803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35495 (local address 127.0.98.3:42135)
0620 04:25:36.373950 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:36.373954 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:36.374177 (+ 223us) server_negotiation.cc:407] Connection header received
0620 04:25:36.374314 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:36.374317 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:36.374368 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:36.374461 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:25:36.374808 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:36.375586 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:36.377339 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:36.377940 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:36.378047 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:36.378118 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:36.378373 (+ 255us) server_negotiation.cc:299] Negotiation successful
0620 04:25:36.378457 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 04:25:36.378597 8499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:36.373713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43319 (local address 127.0.98.2:46855)
0620 04:25:36.374199 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:36.374211 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:36.374288 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:36.374499 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:36.374502 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:36.374513 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:25:36.374704 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:36.374712 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:36.375720 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:36.375723 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:36.377217 (+ 1494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:36.377229 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:36.378087 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:36.378091 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:36.378207 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:36.378240 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 04:25:36.378318 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 04:25:36.378407 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"mutex_wait_us":102,"thread_start_us":60,"threads_started":1}
I20260620 04:25:36.378599 8500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:36.373713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42135 (local address 127.0.98.2:35495)
0620 04:25:36.374061 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:36.374078 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:36.374202 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:36.374478 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:36.374482 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:36.374500 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:25:36.374714 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:36.374719 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:36.375720 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:36.375723 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:36.377217 (+ 1494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:36.377229 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:36.378087 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:36.378091 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:36.378222 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:36.378243 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 04:25:36.378318 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 04:25:36.378407 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"mutex_wait_us":261,"spinlock_wait_cycles":12672,"thread_start_us":73,"threads_started":1}
I20260620 04:25:36.379062 8071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed9ddecf3b5f4b2aac970d1955a6a6fc" candidate_uuid: "f76fa9ea200d4526b6d05f551c0478f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc00f1f932a34415b055f21185d7d3f1" is_pre_election: true
I20260620 04:25:36.379235 8071 raft_consensus.cc:2454] T ed9ddecf3b5f4b2aac970d1955a6a6fc P cc00f1f932a34415b055f21185d7d3f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f76fa9ea200d4526b6d05f551c0478f9 in term 0.
I20260620 04:25:36.379318 8350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed9ddecf3b5f4b2aac970d1955a6a6fc" candidate_uuid: "f76fa9ea200d4526b6d05f551c0478f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6b68ffb3c80489f9a183bfcd20cafca" is_pre_election: true
I20260620 04:25:36.379459 8350 raft_consensus.cc:2454] T ed9ddecf3b5f4b2aac970d1955a6a6fc P b6b68ffb3c80489f9a183bfcd20cafca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f76fa9ea200d4526b6d05f551c0478f9 in term 0.
I20260620 04:25:36.379581 8145 leader_election.cc:302] T ed9ddecf3b5f4b2aac970d1955a6a6fc P f76fa9ea200d4526b6d05f551c0478f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cc00f1f932a34415b055f21185d7d3f1, f76fa9ea200d4526b6d05f551c0478f9; no voters:
I20260620 04:25:36.379946 8432 raft_consensus.cc:2781] T ed9ddecf3b5f4b2aac970d1955a6a6fc P f76fa9ea200d4526b6d05f551c0478f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 04:25:36.380069 8432 raft_consensus.cc:482] T ed9ddecf3b5f4b2aac970d1955a6a6fc P f76fa9ea200d4526b6d05f551c0478f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 04:25:36.380103 8432 raft_consensus.cc:3037] T ed9ddecf3b5f4b2aac970d1955a6a6fc P f76fa9ea200d4526b6d05f551c0478f9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:25:36.380983 8432 raft_consensus.cc:504] T ed9ddecf3b5f4b2aac970d1955a6a6fc P f76fa9ea200d4526b6d05f551c0478f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc00f1f932a34415b055f21185d7d3f1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43319 } } peers { permanent_uuid: "f76fa9ea200d4526b6d05f551c0478f9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45413 } } peers { permanent_uuid: "b6b68ffb3c80489f9a183bfcd20cafca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42135 } }
I20260620 04:25:36.381301 8432 leader_election.cc:288] T ed9ddecf3b5f4b2aac970d1955a6a6fc P f76fa9ea200d4526b6d05f551c0478f9 [CANDIDATE]: Term 1 election: Requested vote from peers cc00f1f932a34415b055f21185d7d3f1 (127.0.98.1:43319), b6b68ffb3c80489f9a183bfcd20cafca (127.0.98.3:42135)
I20260620 04:25:36.381631 8071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed9ddecf3b5f4b2aac970d1955a6a6fc" candidate_uuid: "f76fa9ea200d4526b6d05f551c0478f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc00f1f932a34415b055f21185d7d3f1"
I20260620 04:25:36.381649 8350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed9ddecf3b5f4b2aac970d1955a6a6fc" candidate_uuid: "f76fa9ea200d4526b6d05f551c0478f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6b68ffb3c80489f9a183bfcd20cafca"
I20260620 04:25:36.381759 8350 raft_consensus.cc:3037] T ed9ddecf3b5f4b2aac970d1955a6a6fc P b6b68ffb3c80489f9a183bfcd20cafca [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:25:36.381760 8071 raft_consensus.cc:3037] T ed9ddecf3b5f4b2aac970d1955a6a6fc P cc00f1f932a34415b055f21185d7d3f1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:25:36.382814 8350 raft_consensus.cc:2454] T ed9ddecf3b5f4b2aac970d1955a6a6fc P b6b68ffb3c80489f9a183bfcd20cafca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f76fa9ea200d4526b6d05f551c0478f9 in term 1.
I20260620 04:25:36.382814 8071 raft_consensus.cc:2454] T ed9ddecf3b5f4b2aac970d1955a6a6fc P cc00f1f932a34415b055f21185d7d3f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f76fa9ea200d4526b6d05f551c0478f9 in term 1.
I20260620 04:25:36.383114 8144 leader_election.cc:302] T ed9ddecf3b5f4b2aac970d1955a6a6fc P f76fa9ea200d4526b6d05f551c0478f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: b6b68ffb3c80489f9a183bfcd20cafca, cc00f1f932a34415b055f21185d7d3f1, f76fa9ea200d4526b6d05f551c0478f9; no voters:
I20260620 04:25:36.383277 8432 raft_consensus.cc:2781] T ed9ddecf3b5f4b2aac970d1955a6a6fc P f76fa9ea200d4526b6d05f551c0478f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:25:36.383483 8432 raft_consensus.cc:686] T ed9ddecf3b5f4b2aac970d1955a6a6fc P f76fa9ea200d4526b6d05f551c0478f9 [term 1 LEADER]: Becoming Leader. State: Replica: f76fa9ea200d4526b6d05f551c0478f9, State: Running, Role: LEADER
I20260620 04:25:36.383731 8432 consensus_queue.cc:237] T ed9ddecf3b5f4b2aac970d1955a6a6fc P f76fa9ea200d4526b6d05f551c0478f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc00f1f932a34415b055f21185d7d3f1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43319 } } peers { permanent_uuid: "f76fa9ea200d4526b6d05f551c0478f9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45413 } } peers { permanent_uuid: "b6b68ffb3c80489f9a183bfcd20cafca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42135 } }
I20260620 04:25:36.386049 7913 catalog_manager.cc:5495] T ed9ddecf3b5f4b2aac970d1955a6a6fc P f76fa9ea200d4526b6d05f551c0478f9 reported cstate change: term changed from 0 to 1, leader changed from <none> to f76fa9ea200d4526b6d05f551c0478f9 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "f76fa9ea200d4526b6d05f551c0478f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc00f1f932a34415b055f21185d7d3f1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f76fa9ea200d4526b6d05f551c0478f9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45413 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6b68ffb3c80489f9a183bfcd20cafca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42135 } health_report { overall_health: UNKNOWN } } }
I20260620 04:25:36.405558 8504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:36.402007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49964 (local address 127.0.98.2:45413)
0620 04:25:36.402263 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:36.402270 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:36.402281 (+ 11us) server_negotiation.cc:407] Connection header received
0620 04:25:36.402351 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:36.402353 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:36.402404 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:36.402504 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:25:36.402704 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:36.403254 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:36.404131 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:36.404367 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:36.404454 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:36.404728 (+ 274us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:25:36.405174 (+ 446us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:25:36.405282 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:36.405351 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 04:25:36.405404 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":91,"threads_started":1}
I20260620 04:25:36.408097 8350 raft_consensus.cc:1261] T ed9ddecf3b5f4b2aac970d1955a6a6fc P b6b68ffb3c80489f9a183bfcd20cafca [term 1 FOLLOWER]: Refusing update from remote peer f76fa9ea200d4526b6d05f551c0478f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:25:36.408272 8071 raft_consensus.cc:1261] T ed9ddecf3b5f4b2aac970d1955a6a6fc P cc00f1f932a34415b055f21185d7d3f1 [term 1 FOLLOWER]: Refusing update from remote peer f76fa9ea200d4526b6d05f551c0478f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:25:36.408560 8432 consensus_queue.cc:1035] T ed9ddecf3b5f4b2aac970d1955a6a6fc P f76fa9ea200d4526b6d05f551c0478f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6b68ffb3c80489f9a183bfcd20cafca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:25:36.408766 8501 consensus_queue.cc:1035] T ed9ddecf3b5f4b2aac970d1955a6a6fc P f76fa9ea200d4526b6d05f551c0478f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc00f1f932a34415b055f21185d7d3f1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:25:36.420199 7912 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35862:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:25:36.420332 7912 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35862:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:25:36.422104 7912 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ac9ebe4a2398411f93639541ab97f9e7: Sending DeleteTablet for 3 replicas of tablet ed9ddecf3b5f4b2aac970d1955a6a6fc
I20260620 04:25:36.422667 8331 tablet_service.cc:1514] Processing DeleteTablet for tablet ed9ddecf3b5f4b2aac970d1955a6a6fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:25:36 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60246
I20260620 04:25:36.422681 8051 tablet_service.cc:1514] Processing DeleteTablet for tablet ed9ddecf3b5f4b2aac970d1955a6a6fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:25:36 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33054
I20260620 04:25:36.422839 8448 tablet_replica.cc:330] T ed9ddecf3b5f4b2aac970d1955a6a6fc P b6b68ffb3c80489f9a183bfcd20cafca: stopping tablet replica
I20260620 04:25:36.422875 8449 tablet_replica.cc:330] T ed9ddecf3b5f4b2aac970d1955a6a6fc P cc00f1f932a34415b055f21185d7d3f1: stopping tablet replica
I20260620 04:25:36.422952 8448 raft_consensus.cc:2229] T ed9ddecf3b5f4b2aac970d1955a6a6fc P b6b68ffb3c80489f9a183bfcd20cafca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:25:36.422987 8449 raft_consensus.cc:2229] T ed9ddecf3b5f4b2aac970d1955a6a6fc P cc00f1f932a34415b055f21185d7d3f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:25:36.423076 8448 raft_consensus.cc:2258] T ed9ddecf3b5f4b2aac970d1955a6a6fc P b6b68ffb3c80489f9a183bfcd20cafca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:25:36.423110 8449 raft_consensus.cc:2258] T ed9ddecf3b5f4b2aac970d1955a6a6fc P cc00f1f932a34415b055f21185d7d3f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:25:36.423866 8449 ts_tablet_manager.cc:1905] T ed9ddecf3b5f4b2aac970d1955a6a6fc P cc00f1f932a34415b055f21185d7d3f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:25:36.423942 7912 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35862:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 04:25:36.424196 8448 ts_tablet_manager.cc:1905] T ed9ddecf3b5f4b2aac970d1955a6a6fc P b6b68ffb3c80489f9a183bfcd20cafca: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 04:25:36.424376 7912 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 04:25:36.426257 8448 ts_tablet_manager.cc:1918] T ed9ddecf3b5f4b2aac970d1955a6a6fc P b6b68ffb3c80489f9a183bfcd20cafca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:25:36.426323 8448 log.cc:1196] T ed9ddecf3b5f4b2aac970d1955a6a6fc P b6b68ffb3c80489f9a183bfcd20cafca: Deleting WAL directory at /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929496330398-392-0/minicluster-data/ts-2/wal/wals/ed9ddecf3b5f4b2aac970d1955a6a6fc
I20260620 04:25:36.426383 8449 ts_tablet_manager.cc:1918] T ed9ddecf3b5f4b2aac970d1955a6a6fc P cc00f1f932a34415b055f21185d7d3f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:25:36.426466 8449 log.cc:1196] T ed9ddecf3b5f4b2aac970d1955a6a6fc P cc00f1f932a34415b055f21185d7d3f1: Deleting WAL directory at /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929496330398-392-0/minicluster-data/ts-0/wal/wals/ed9ddecf3b5f4b2aac970d1955a6a6fc
I20260620 04:25:36.426566 8448 ts_tablet_manager.cc:1939] T ed9ddecf3b5f4b2aac970d1955a6a6fc P b6b68ffb3c80489f9a183bfcd20cafca: Deleting consensus metadata
I20260620 04:25:36.426730 8449 ts_tablet_manager.cc:1939] T ed9ddecf3b5f4b2aac970d1955a6a6fc P cc00f1f932a34415b055f21185d7d3f1: Deleting consensus metadata
I20260620 04:25:36.427143 7900 catalog_manager.cc:4841] TS b6b68ffb3c80489f9a183bfcd20cafca (127.0.98.3:42135): tablet ed9ddecf3b5f4b2aac970d1955a6a6fc (table test-table [id=6064e6a340bf4309ae980c790db4a755]) successfully deleted
I20260620 04:25:36.427752 8191 tablet_service.cc:1514] Processing DeleteTablet for tablet ed9ddecf3b5f4b2aac970d1955a6a6fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:25:36 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49960
I20260620 04:25:36.427940 8450 tablet_replica.cc:330] T ed9ddecf3b5f4b2aac970d1955a6a6fc P f76fa9ea200d4526b6d05f551c0478f9: stopping tablet replica
I20260620 04:25:36.428064 8450 raft_consensus.cc:2229] T ed9ddecf3b5f4b2aac970d1955a6a6fc P f76fa9ea200d4526b6d05f551c0478f9 [term 1 LEADER]: Raft consensus shutting down.
I20260620 04:25:36.428248 7901 catalog_manager.cc:4841] TS cc00f1f932a34415b055f21185d7d3f1 (127.0.98.1:43319): tablet ed9ddecf3b5f4b2aac970d1955a6a6fc (table test-table [id=6064e6a340bf4309ae980c790db4a755]) successfully deleted
I20260620 04:25:36.428344 8450 raft_consensus.cc:2258] T ed9ddecf3b5f4b2aac970d1955a6a6fc P f76fa9ea200d4526b6d05f551c0478f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:25:36.428902 8450 ts_tablet_manager.cc:1905] T ed9ddecf3b5f4b2aac970d1955a6a6fc P f76fa9ea200d4526b6d05f551c0478f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:25:36.431227 8450 ts_tablet_manager.cc:1918] T ed9ddecf3b5f4b2aac970d1955a6a6fc P f76fa9ea200d4526b6d05f551c0478f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:25:36.431298 8450 log.cc:1196] T ed9ddecf3b5f4b2aac970d1955a6a6fc P f76fa9ea200d4526b6d05f551c0478f9: Deleting WAL directory at /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929496330398-392-0/minicluster-data/ts-1/wal/wals/ed9ddecf3b5f4b2aac970d1955a6a6fc
I20260620 04:25:36.431382 8051 tablet_service.cc:1467] Processing CreateTablet for tablet a4f3444bb1324500ad0e8f869e0495f7 (DEFAULT_TABLE table=test-table [id=d3df634ea125414480d1c68b8afd1f43]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:25:36.431526 8450 ts_tablet_manager.cc:1939] T ed9ddecf3b5f4b2aac970d1955a6a6fc P f76fa9ea200d4526b6d05f551c0478f9: Deleting consensus metadata
I20260620 04:25:36.431545 8331 tablet_service.cc:1467] Processing CreateTablet for tablet a4f3444bb1324500ad0e8f869e0495f7 (DEFAULT_TABLE table=test-table [id=d3df634ea125414480d1c68b8afd1f43]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:25:36.431627 8051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4f3444bb1324500ad0e8f869e0495f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:25:36.431758 8331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4f3444bb1324500ad0e8f869e0495f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:25:36.433435 8426 tablet_bootstrap.cc:492] T a4f3444bb1324500ad0e8f869e0495f7 P b6b68ffb3c80489f9a183bfcd20cafca: Bootstrap starting.
I20260620 04:25:36.433840 8425 tablet_bootstrap.cc:492] T a4f3444bb1324500ad0e8f869e0495f7 P cc00f1f932a34415b055f21185d7d3f1: Bootstrap starting.
I20260620 04:25:36.434191 8426 tablet_bootstrap.cc:654] T a4f3444bb1324500ad0e8f869e0495f7 P b6b68ffb3c80489f9a183bfcd20cafca: Neither blocks nor log segments found. Creating new log.
I20260620 04:25:36.434728 8425 tablet_bootstrap.cc:654] T a4f3444bb1324500ad0e8f869e0495f7 P cc00f1f932a34415b055f21185d7d3f1: Neither blocks nor log segments found. Creating new log.
I20260620 04:25:36.435532 8191 tablet_service.cc:1467] Processing CreateTablet for tablet a4f3444bb1324500ad0e8f869e0495f7 (DEFAULT_TABLE table=test-table [id=d3df634ea125414480d1c68b8afd1f43]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:25:36.435726 8425 tablet_bootstrap.cc:492] T a4f3444bb1324500ad0e8f869e0495f7 P cc00f1f932a34415b055f21185d7d3f1: No bootstrap required, opened a new log
I20260620 04:25:36.435761 8191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4f3444bb1324500ad0e8f869e0495f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:25:36.435792 8425 ts_tablet_manager.cc:1397] T a4f3444bb1324500ad0e8f869e0495f7 P cc00f1f932a34415b055f21185d7d3f1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 04:25:36.435902 8426 tablet_bootstrap.cc:492] T a4f3444bb1324500ad0e8f869e0495f7 P b6b68ffb3c80489f9a183bfcd20cafca: No bootstrap required, opened a new log
I20260620 04:25:36.435992 8426 ts_tablet_manager.cc:1397] T a4f3444bb1324500ad0e8f869e0495f7 P b6b68ffb3c80489f9a183bfcd20cafca: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260620 04:25:36.436193 8425 raft_consensus.cc:348] T a4f3444bb1324500ad0e8f869e0495f7 P cc00f1f932a34415b055f21185d7d3f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc00f1f932a34415b055f21185d7d3f1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43319 } } peers { permanent_uuid: "f76fa9ea200d4526b6d05f551c0478f9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45413 } } peers { permanent_uuid: "b6b68ffb3c80489f9a183bfcd20cafca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42135 } }
I20260620 04:25:36.436309 8425 raft_consensus.cc:374] T a4f3444bb1324500ad0e8f869e0495f7 P cc00f1f932a34415b055f21185d7d3f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:25:36.436337 8425 raft_consensus.cc:729] T a4f3444bb1324500ad0e8f869e0495f7 P cc00f1f932a34415b055f21185d7d3f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc00f1f932a34415b055f21185d7d3f1, State: Initialized, Role: FOLLOWER
I20260620 04:25:36.436371 8426 raft_consensus.cc:348] T a4f3444bb1324500ad0e8f869e0495f7 P b6b68ffb3c80489f9a183bfcd20cafca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc00f1f932a34415b055f21185d7d3f1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43319 } } peers { permanent_uuid: "f76fa9ea200d4526b6d05f551c0478f9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45413 } } peers { permanent_uuid: "b6b68ffb3c80489f9a183bfcd20cafca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42135 } }
I20260620 04:25:36.436470 8426 raft_consensus.cc:374] T a4f3444bb1324500ad0e8f869e0495f7 P b6b68ffb3c80489f9a183bfcd20cafca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:25:36.436502 8426 raft_consensus.cc:729] T a4f3444bb1324500ad0e8f869e0495f7 P b6b68ffb3c80489f9a183bfcd20cafca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6b68ffb3c80489f9a183bfcd20cafca, State: Initialized, Role: FOLLOWER
I20260620 04:25:36.436462 8425 consensus_queue.cc:260] T a4f3444bb1324500ad0e8f869e0495f7 P cc00f1f932a34415b055f21185d7d3f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc00f1f932a34415b055f21185d7d3f1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43319 } } peers { permanent_uuid: "f76fa9ea200d4526b6d05f551c0478f9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45413 } } peers { permanent_uuid: "b6b68ffb3c80489f9a183bfcd20cafca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42135 } }
I20260620 04:25:36.436615 8426 consensus_queue.cc:260] T a4f3444bb1324500ad0e8f869e0495f7 P b6b68ffb3c80489f9a183bfcd20cafca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc00f1f932a34415b055f21185d7d3f1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43319 } } peers { permanent_uuid: "f76fa9ea200d4526b6d05f551c0478f9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45413 } } peers { permanent_uuid: "b6b68ffb3c80489f9a183bfcd20cafca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42135 } }
I20260620 04:25:36.436722 8425 ts_tablet_manager.cc:1428] T a4f3444bb1324500ad0e8f869e0495f7 P cc00f1f932a34415b055f21185d7d3f1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 04:25:36.436848 8426 ts_tablet_manager.cc:1428] T a4f3444bb1324500ad0e8f869e0495f7 P b6b68ffb3c80489f9a183bfcd20cafca: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 04:25:36.437863 8427 tablet_bootstrap.cc:492] T a4f3444bb1324500ad0e8f869e0495f7 P f76fa9ea200d4526b6d05f551c0478f9: Bootstrap starting.
I20260620 04:25:36.438282 7901 catalog_manager.cc:4841] TS f76fa9ea200d4526b6d05f551c0478f9 (127.0.98.2:45413): tablet ed9ddecf3b5f4b2aac970d1955a6a6fc (table test-table [id=6064e6a340bf4309ae980c790db4a755]) successfully deleted
I20260620 04:25:36.438705 8427 tablet_bootstrap.cc:654] T a4f3444bb1324500ad0e8f869e0495f7 P f76fa9ea200d4526b6d05f551c0478f9: Neither blocks nor log segments found. Creating new log.
I20260620 04:25:36.440685 8427 tablet_bootstrap.cc:492] T a4f3444bb1324500ad0e8f869e0495f7 P f76fa9ea200d4526b6d05f551c0478f9: No bootstrap required, opened a new log
I20260620 04:25:36.440768 8427 ts_tablet_manager.cc:1397] T a4f3444bb1324500ad0e8f869e0495f7 P f76fa9ea200d4526b6d05f551c0478f9: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260620 04:25:36.441218 8427 raft_consensus.cc:348] T a4f3444bb1324500ad0e8f869e0495f7 P f76fa9ea200d4526b6d05f551c0478f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc00f1f932a34415b055f21185d7d3f1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43319 } } peers { permanent_uuid: "f76fa9ea200d4526b6d05f551c0478f9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45413 } } peers { permanent_uuid: "b6b68ffb3c80489f9a183bfcd20cafca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42135 } }
I20260620 04:25:36.441329 8427 raft_consensus.cc:374] T a4f3444bb1324500ad0e8f869e0495f7 P f76fa9ea200d4526b6d05f551c0478f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:25:36.441359 8427 raft_consensus.cc:729] T a4f3444bb1324500ad0e8f869e0495f7 P f76fa9ea200d4526b6d05f551c0478f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f76fa9ea200d4526b6d05f551c0478f9, State: Initialized, Role: FOLLOWER
I20260620 04:25:36.441475 8427 consensus_queue.cc:260] T a4f3444bb1324500ad0e8f869e0495f7 P f76fa9ea200d4526b6d05f551c0478f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc00f1f932a34415b055f21185d7d3f1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43319 } } peers { permanent_uuid: "f76fa9ea200d4526b6d05f551c0478f9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45413 } } peers { permanent_uuid: "b6b68ffb3c80489f9a183bfcd20cafca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42135 } }
I20260620 04:25:36.441728 8427 ts_tablet_manager.cc:1428] T a4f3444bb1324500ad0e8f869e0495f7 P f76fa9ea200d4526b6d05f551c0478f9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 04:25:36.508446 8432 raft_consensus.cc:482] T a4f3444bb1324500ad0e8f869e0495f7 P f76fa9ea200d4526b6d05f551c0478f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:25:36.508595 8432 raft_consensus.cc:504] T a4f3444bb1324500ad0e8f869e0495f7 P f76fa9ea200d4526b6d05f551c0478f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc00f1f932a34415b055f21185d7d3f1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43319 } } peers { permanent_uuid: "f76fa9ea200d4526b6d05f551c0478f9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45413 } } peers { permanent_uuid: "b6b68ffb3c80489f9a183bfcd20cafca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42135 } }
I20260620 04:25:36.509037 8432 leader_election.cc:288] T a4f3444bb1324500ad0e8f869e0495f7 P f76fa9ea200d4526b6d05f551c0478f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc00f1f932a34415b055f21185d7d3f1 (127.0.98.1:43319), b6b68ffb3c80489f9a183bfcd20cafca (127.0.98.3:42135)
I20260620 04:25:36.509454 8071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4f3444bb1324500ad0e8f869e0495f7" candidate_uuid: "f76fa9ea200d4526b6d05f551c0478f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc00f1f932a34415b055f21185d7d3f1" is_pre_election: true
I20260620 04:25:36.509624 8071 raft_consensus.cc:2454] T a4f3444bb1324500ad0e8f869e0495f7 P cc00f1f932a34415b055f21185d7d3f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f76fa9ea200d4526b6d05f551c0478f9 in term 0.
I20260620 04:25:36.509658 8350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4f3444bb1324500ad0e8f869e0495f7" candidate_uuid: "f76fa9ea200d4526b6d05f551c0478f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6b68ffb3c80489f9a183bfcd20cafca" is_pre_election: true
I20260620 04:25:36.509865 8350 raft_consensus.cc:2454] T a4f3444bb1324500ad0e8f869e0495f7 P b6b68ffb3c80489f9a183bfcd20cafca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f76fa9ea200d4526b6d05f551c0478f9 in term 0.
I20260620 04:25:36.509905 8145 leader_election.cc:302] T a4f3444bb1324500ad0e8f869e0495f7 P f76fa9ea200d4526b6d05f551c0478f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cc00f1f932a34415b055f21185d7d3f1, f76fa9ea200d4526b6d05f551c0478f9; no voters:
I20260620 04:25:36.510070 8432 raft_consensus.cc:2781] T a4f3444bb1324500ad0e8f869e0495f7 P f76fa9ea200d4526b6d05f551c0478f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 04:25:36.510118 8432 raft_consensus.cc:482] T a4f3444bb1324500ad0e8f869e0495f7 P f76fa9ea200d4526b6d05f551c0478f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 04:25:36.510151 8432 raft_consensus.cc:3037] T a4f3444bb1324500ad0e8f869e0495f7 P f76fa9ea200d4526b6d05f551c0478f9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:25:36.511252 8432 raft_consensus.cc:504] T a4f3444bb1324500ad0e8f869e0495f7 P f76fa9ea200d4526b6d05f551c0478f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc00f1f932a34415b055f21185d7d3f1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43319 } } peers { permanent_uuid: "f76fa9ea200d4526b6d05f551c0478f9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45413 } } peers { permanent_uuid: "b6b68ffb3c80489f9a183bfcd20cafca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42135 } }
I20260620 04:25:36.511590 8432 leader_election.cc:288] T a4f3444bb1324500ad0e8f869e0495f7 P f76fa9ea200d4526b6d05f551c0478f9 [CANDIDATE]: Term 1 election: Requested vote from peers cc00f1f932a34415b055f21185d7d3f1 (127.0.98.1:43319), b6b68ffb3c80489f9a183bfcd20cafca (127.0.98.3:42135)
I20260620 04:25:36.511930 8350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4f3444bb1324500ad0e8f869e0495f7" candidate_uuid: "f76fa9ea200d4526b6d05f551c0478f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6b68ffb3c80489f9a183bfcd20cafca"
I20260620 04:25:36.511994 8071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4f3444bb1324500ad0e8f869e0495f7" candidate_uuid: "f76fa9ea200d4526b6d05f551c0478f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc00f1f932a34415b055f21185d7d3f1"
I20260620 04:25:36.512069 8350 raft_consensus.cc:3037] T a4f3444bb1324500ad0e8f869e0495f7 P b6b68ffb3c80489f9a183bfcd20cafca [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:25:36.512128 8071 raft_consensus.cc:3037] T a4f3444bb1324500ad0e8f869e0495f7 P cc00f1f932a34415b055f21185d7d3f1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:25:36.512986 8350 raft_consensus.cc:2454] T a4f3444bb1324500ad0e8f869e0495f7 P b6b68ffb3c80489f9a183bfcd20cafca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f76fa9ea200d4526b6d05f551c0478f9 in term 1.
I20260620 04:25:36.513244 8144 leader_election.cc:302] T a4f3444bb1324500ad0e8f869e0495f7 P f76fa9ea200d4526b6d05f551c0478f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b6b68ffb3c80489f9a183bfcd20cafca, f76fa9ea200d4526b6d05f551c0478f9; no voters:
I20260620 04:25:36.513418 8432 raft_consensus.cc:2781] T a4f3444bb1324500ad0e8f869e0495f7 P f76fa9ea200d4526b6d05f551c0478f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:25:36.513386 8071 raft_consensus.cc:2454] T a4f3444bb1324500ad0e8f869e0495f7 P cc00f1f932a34415b055f21185d7d3f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f76fa9ea200d4526b6d05f551c0478f9 in term 1.
I20260620 04:25:36.513489 8432 raft_consensus.cc:686] T a4f3444bb1324500ad0e8f869e0495f7 P f76fa9ea200d4526b6d05f551c0478f9 [term 1 LEADER]: Becoming Leader. State: Replica: f76fa9ea200d4526b6d05f551c0478f9, State: Running, Role: LEADER
I20260620 04:25:36.513614 8432 consensus_queue.cc:237] T a4f3444bb1324500ad0e8f869e0495f7 P f76fa9ea200d4526b6d05f551c0478f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc00f1f932a34415b055f21185d7d3f1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43319 } } peers { permanent_uuid: "f76fa9ea200d4526b6d05f551c0478f9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45413 } } peers { permanent_uuid: "b6b68ffb3c80489f9a183bfcd20cafca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42135 } }
I20260620 04:25:36.515357 7912 catalog_manager.cc:5495] T a4f3444bb1324500ad0e8f869e0495f7 P f76fa9ea200d4526b6d05f551c0478f9 reported cstate change: term changed from 0 to 1, leader changed from <none> to f76fa9ea200d4526b6d05f551c0478f9 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "f76fa9ea200d4526b6d05f551c0478f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc00f1f932a34415b055f21185d7d3f1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f76fa9ea200d4526b6d05f551c0478f9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45413 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6b68ffb3c80489f9a183bfcd20cafca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42135 } health_report { overall_health: UNKNOWN } } }
I20260620 04:25:36.555058 8421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:36.551244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:32869 (local address 127.0.98.3:42135)
0620 04:25:36.551369 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:36.551374 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:36.551475 (+ 101us) server_negotiation.cc:407] Connection header received
0620 04:25:36.551606 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:36.551609 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:36.551660 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:36.551731 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:25:36.552089 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:36.552984 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:36.553944 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:36.554278 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:36.554377 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:36.554623 (+ 246us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:25:36.554707 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:25:36.554772 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:36.554835 (+ 63us) server_negotiation.cc:299] Negotiation successful
0620 04:25:36.554888 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 04:25:36.555043 8517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:36.551096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42135 (local address 127.0.98.2:32869)
0620 04:25:36.551393 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:36.551408 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:36.551515 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:36.551770 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:36.551772 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:36.551780 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 04:25:36.551908 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:36.551913 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:36.553151 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:36.553154 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:36.553815 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:36.553823 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:36.554388 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:36.554390 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:36.554464 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:36.554511 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 04:25:36.554776 (+ 265us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 04:25:36.554780 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 04:25:36.554822 (+ 42us) client_negotiation.cc:240] Negotiation successful
0620 04:25:36.554885 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":79,"threads_started":1}
I20260620 04:25:36.561338 8350 raft_consensus.cc:1261] T a4f3444bb1324500ad0e8f869e0495f7 P b6b68ffb3c80489f9a183bfcd20cafca [term 1 FOLLOWER]: Refusing update from remote peer f76fa9ea200d4526b6d05f551c0478f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:25:36.561358 8071 raft_consensus.cc:1261] T a4f3444bb1324500ad0e8f869e0495f7 P cc00f1f932a34415b055f21185d7d3f1 [term 1 FOLLOWER]: Refusing update from remote peer f76fa9ea200d4526b6d05f551c0478f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:25:36.561717 8507 consensus_queue.cc:1035] T a4f3444bb1324500ad0e8f869e0495f7 P f76fa9ea200d4526b6d05f551c0478f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6b68ffb3c80489f9a183bfcd20cafca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:25:36.561861 8432 consensus_queue.cc:1035] T a4f3444bb1324500ad0e8f869e0495f7 P f76fa9ea200d4526b6d05f551c0478f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc00f1f932a34415b055f21185d7d3f1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:25:36.580021 8504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:36.576853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55323 (local address 127.0.98.2:45413)
0620 04:25:36.576966 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:36.576970 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:36.577102 (+ 132us) server_negotiation.cc:407] Connection header received
0620 04:25:36.577224 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:36.577227 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:36.577272 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:36.577330 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:25:36.577563 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:36.578112 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:36.578916 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:36.579169 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:36.579264 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:36.579525 (+ 261us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:25:36.579605 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:25:36.579676 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:36.579790 (+ 114us) server_negotiation.cc:299] Negotiation successful
0620 04:25:36.579849 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 04:25:36.580041 8521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:36.576735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45413 (local address 127.0.98.3:55323)
0620 04:25:36.577012 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:36.577027 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:36.577124 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:36.577346 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:36.577349 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:36.577356 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 04:25:36.577452 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:36.577457 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:36.578226 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:36.578228 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:36.578814 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:36.578820 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:36.579265 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:36.579268 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:36.579374 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:36.579408 (+ 34us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 04:25:36.579695 (+ 287us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 04:25:36.579701 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 04:25:36.579777 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 04:25:36.579869 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":85,"threads_started":1}
I20260620 04:25:36.600265 7911 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35862:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:25:36.600418 7911 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35862:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:25:36.602322 7911 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ac9ebe4a2398411f93639541ab97f9e7: Sending DeleteTablet for 3 replicas of tablet a4f3444bb1324500ad0e8f869e0495f7
I20260620 04:25:36.602865 8191 tablet_service.cc:1514] Processing DeleteTablet for tablet a4f3444bb1324500ad0e8f869e0495f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:25:36 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49960
I20260620 04:25:36.602950 8331 tablet_service.cc:1514] Processing DeleteTablet for tablet a4f3444bb1324500ad0e8f869e0495f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:25:36 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60246
I20260620 04:25:36.602984 8051 tablet_service.cc:1514] Processing DeleteTablet for tablet a4f3444bb1324500ad0e8f869e0495f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:25:36 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33054
I20260620 04:25:36.603044 8450 tablet_replica.cc:330] T a4f3444bb1324500ad0e8f869e0495f7 P f76fa9ea200d4526b6d05f551c0478f9: stopping tablet replica
I20260620 04:25:36.603130 8448 tablet_replica.cc:330] T a4f3444bb1324500ad0e8f869e0495f7 P b6b68ffb3c80489f9a183bfcd20cafca: stopping tablet replica
I20260620 04:25:36.603142 8450 raft_consensus.cc:2229] T a4f3444bb1324500ad0e8f869e0495f7 P f76fa9ea200d4526b6d05f551c0478f9 [term 1 LEADER]: Raft consensus shutting down.
I20260620 04:25:36.603223 8449 tablet_replica.cc:330] T a4f3444bb1324500ad0e8f869e0495f7 P cc00f1f932a34415b055f21185d7d3f1: stopping tablet replica
I20260620 04:25:36.603247 8448 raft_consensus.cc:2229] T a4f3444bb1324500ad0e8f869e0495f7 P b6b68ffb3c80489f9a183bfcd20cafca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:25:36.603336 8449 raft_consensus.cc:2229] T a4f3444bb1324500ad0e8f869e0495f7 P cc00f1f932a34415b055f21185d7d3f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:25:36.603360 8448 raft_consensus.cc:2258] T a4f3444bb1324500ad0e8f869e0495f7 P b6b68ffb3c80489f9a183bfcd20cafca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:25:36.603363 8450 raft_consensus.cc:2258] T a4f3444bb1324500ad0e8f869e0495f7 P f76fa9ea200d4526b6d05f551c0478f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:25:36.603442 8449 raft_consensus.cc:2258] T a4f3444bb1324500ad0e8f869e0495f7 P cc00f1f932a34415b055f21185d7d3f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:25:36.603996 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 7986
I20260620 04:25:36.604089 8449 ts_tablet_manager.cc:1905] T a4f3444bb1324500ad0e8f869e0495f7 P cc00f1f932a34415b055f21185d7d3f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:25:36.604089 8450 ts_tablet_manager.cc:1905] T a4f3444bb1324500ad0e8f869e0495f7 P f76fa9ea200d4526b6d05f551c0478f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:25:36.604113 8448 ts_tablet_manager.cc:1905] T a4f3444bb1324500ad0e8f869e0495f7 P b6b68ffb3c80489f9a183bfcd20cafca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:25:36.605974 8450 ts_tablet_manager.cc:1918] T a4f3444bb1324500ad0e8f869e0495f7 P f76fa9ea200d4526b6d05f551c0478f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 04:25:36.606024 8450 log.cc:1196] T a4f3444bb1324500ad0e8f869e0495f7 P f76fa9ea200d4526b6d05f551c0478f9: Deleting WAL directory at /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929496330398-392-0/minicluster-data/ts-1/wal/wals/a4f3444bb1324500ad0e8f869e0495f7
I20260620 04:25:36.606209 8450 ts_tablet_manager.cc:1939] T a4f3444bb1324500ad0e8f869e0495f7 P f76fa9ea200d4526b6d05f551c0478f9: Deleting consensus metadata
I20260620 04:25:36.606403 8448 ts_tablet_manager.cc:1918] T a4f3444bb1324500ad0e8f869e0495f7 P b6b68ffb3c80489f9a183bfcd20cafca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 04:25:36.606467 8448 log.cc:1196] T a4f3444bb1324500ad0e8f869e0495f7 P b6b68ffb3c80489f9a183bfcd20cafca: Deleting WAL directory at /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929496330398-392-0/minicluster-data/ts-2/wal/wals/a4f3444bb1324500ad0e8f869e0495f7
I20260620 04:25:36.606681 8448 ts_tablet_manager.cc:1939] T a4f3444bb1324500ad0e8f869e0495f7 P b6b68ffb3c80489f9a183bfcd20cafca: Deleting consensus metadata
I20260620 04:25:36.607003 7901 catalog_manager.cc:4841] TS f76fa9ea200d4526b6d05f551c0478f9 (127.0.98.2:45413): tablet a4f3444bb1324500ad0e8f869e0495f7 (table test-table [id=d3df634ea125414480d1c68b8afd1f43]) successfully deleted
I20260620 04:25:36.608513 7900 catalog_manager.cc:4841] TS b6b68ffb3c80489f9a183bfcd20cafca (127.0.98.3:42135): tablet a4f3444bb1324500ad0e8f869e0495f7 (table test-table [id=d3df634ea125414480d1c68b8afd1f43]) successfully deleted
W20260620 04:25:36.612188 7901 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43319 (error 108)
I20260620 04:25:36.612341 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 8126
I20260620 04:25:36.614123 8524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:36.613567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43319 (local address 127.0.0.1:33076)
0620 04:25:36.613964 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:36.614036 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":53,"threads_started":1}
W20260620 04:25:36.614264 7901 catalog_manager.cc:4584] TS cc00f1f932a34415b055f21185d7d3f1 (127.0.98.1:43319): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a4f3444bb1324500ad0e8f869e0495f7: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43319: connect: Connection refused (error 111)
I20260620 04:25:36.618326 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 8266
I20260620 04:25:36.623994 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 7882
2026-06-20T04:25:36Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3349 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781929496330398-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:25:36 dist-test-slave-nk6d krb5kdc[8527](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:25:36 dist-test-slave-nk6d krb5kdc[8527](info): set up 2 sockets
Jun 20 04:25:36 dist-test-slave-nk6d krb5kdc[8527](info): commencing operation
krb5kdc: starting...
W20260620 04:25:37.576449 392 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781929496330398-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781929496330398-392-0/krb5kdc/test-user.keytab.
Jun 20 04:25:37 dist-test-slave-nk6d krb5kdc[8527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929537, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:25:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:25:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 04:25:37.688771 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781929496330398-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781929496330398-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781929496330398-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41125
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41125
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:25:37.794070 8540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:25:37.794346 8540 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:25:37.794404 8540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:25:37.798403 8540 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:25:37.798458 8540 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:25:37.798484 8540 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:25:37.798506 8540 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:25:37.798525 8540 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:25:37.803690 8540 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781929496330398-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781929496330398-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41125
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41125
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8540
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781929496330398-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:25:37.804706 8540 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:25:37.805691 8540 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:25:37.813683 8545 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:25:37.813694 8546 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:25:37.813797 8540 server_base.cc:1034] running on GCE node
W20260620 04:25:37.813694 8548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:25:37.814476 8540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:25:37.815554 8540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:25:37.816727 8540 hybrid_clock.cc:648] HybridClock initialized: now 1781929537816711 us; error 30 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-20T04:25:37Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1202 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:25:37 dist-test-slave-nk6d krb5kdc[8553](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:25:37 dist-test-slave-nk6d krb5kdc[8553](info): set up 2 sockets
Jun 20 04:25:37 dist-test-slave-nk6d krb5kdc[8553](info): commencing operation
krb5kdc: starting...
W20260620 04:25:38.776336 392 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/krb5kdc/test-user.keytab.
Jun 20 04:25:38 dist-test-slave-nk6d krb5kdc[8553](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929538, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:25:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:25:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 04:25:38.888195 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36971
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36971
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:25:38.993482 8566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:25:38.993748 8566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:25:38.993786 8566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:25:38.997733 8566 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:25:38.997790 8566 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:25:38.997808 8566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:25:38.997824 8566 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:25:38.997838 8566 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:25:39.002965 8566 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36971
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36971
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8566
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:25:39.003907 8566 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:25:39.004907 8566 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:25:39.013058 8574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:25:39.013074 8571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:25:39.013159 8566 server_base.cc:1034] running on GCE node
W20260620 04:25:39.013139 8572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:25:39.013840 8566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:25:39.014879 8566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:25:39.016016 8566 hybrid_clock.cc:648] HybridClock initialized: now 1781929539015996 us; error 28 us; skew 500 ppm
Jun 20 04:25:39 dist-test-slave-nk6d krb5kdc[8553](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929539, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:25:39.018514 8566 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 04:25:39.019387 8566 webserver.cc:460] Webserver started at http://127.0.98.62:42025/ using document root <none> and password file <none>
I20260620 04:25:39.019991 8566 fs_manager.cc:359] Metadata directory not provided
I20260620 04:25:39.020063 8566 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:25:39.020255 8566 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:25:39.022255 8566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/master-0/data/instance:
uuid: "2345fcf611134eb7bfe8db60c8aa95a6"
format_stamp: "Formatted at 2026-06-20 04:25:39 on dist-test-slave-nk6d"
server_key: "994ed8c2bb27006c49af45cb4f281b12"
server_key_iv: "ee254f859d171fb5383d6b2c0ff4a39b"
server_key_version: "encryptionkey@0"
I20260620 04:25:39.022782 8566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/master-0/wal/instance:
uuid: "2345fcf611134eb7bfe8db60c8aa95a6"
format_stamp: "Formatted at 2026-06-20 04:25:39 on dist-test-slave-nk6d"
server_key: "994ed8c2bb27006c49af45cb4f281b12"
server_key_iv: "ee254f859d171fb5383d6b2c0ff4a39b"
server_key_version: "encryptionkey@0"
I20260620 04:25:39.026414 8566 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 04:25:39.028731 8581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:39.030026 8566 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:25:39.030172 8566 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/master-0/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/master-0/wal
uuid: "2345fcf611134eb7bfe8db60c8aa95a6"
format_stamp: "Formatted at 2026-06-20 04:25:39 on dist-test-slave-nk6d"
server_key: "994ed8c2bb27006c49af45cb4f281b12"
server_key_iv: "ee254f859d171fb5383d6b2c0ff4a39b"
server_key_version: "encryptionkey@0"
I20260620 04:25:39.030278 8566 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:25:39.060667 8566 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:25:39.063674 8566 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:25:39.063853 8566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:25:39.070792 8566 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36971
I20260620 04:25:39.070861 8633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36971 every 8 connection(s)
I20260620 04:25:39.071758 8566 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:25:39.074221 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 8566
I20260620 04:25:39.074348 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/master-0/wal/instance
I20260620 04:25:39.074688 392 external_mini_cluster.cc:1428] Setting key b364f2e8910d2a4663856fe165023138
I20260620 04:25:39.074759 8634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:25:39.081857 8634 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2345fcf611134eb7bfe8db60c8aa95a6: Bootstrap starting.
Jun 20 04:25:39 dist-test-slave-nk6d krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929538, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:25:39.084690 8634 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2345fcf611134eb7bfe8db60c8aa95a6: Neither blocks nor log segments found. Creating new log.
I20260620 04:25:39.085436 8634 log.cc:824] T 00000000000000000000000000000000 P 2345fcf611134eb7bfe8db60c8aa95a6: Log is configured to *not* fsync() on all Append() calls
I20260620 04:25:39.087271 8634 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2345fcf611134eb7bfe8db60c8aa95a6: No bootstrap required, opened a new log
I20260620 04:25:39.089243 8637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:39.076147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40142 (local address 127.0.98.62:36971)
0620 04:25:39.076555 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:39.076571 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:39.076606 (+ 35us) server_negotiation.cc:407] Connection header received
0620 04:25:39.077383 (+ 777us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:39.077402 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:39.077735 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:39.078028 (+ 293us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:39.078921 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:39.080014 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:39.081346 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:39.081856 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:39.082040 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:39.084906 (+ 2866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:39.084912 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:39.084926 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:25:39.084960 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:39.086582 (+ 1622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:39.086992 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:39.086998 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:39.087004 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:39.087113 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:39.087360 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:39.087363 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:39.087365 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:39.087779 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:39.087947 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:39.088271 (+ 324us) server_negotiation.cc:299] Negotiation successful
0620 04:25:39.088552 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":129,"threads_started":1}
I20260620 04:25:39.090996 8634 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2345fcf611134eb7bfe8db60c8aa95a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2345fcf611134eb7bfe8db60c8aa95a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36971 } }
I20260620 04:25:39.091260 8634 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2345fcf611134eb7bfe8db60c8aa95a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:25:39.091344 8634 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2345fcf611134eb7bfe8db60c8aa95a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2345fcf611134eb7bfe8db60c8aa95a6, State: Initialized, Role: FOLLOWER
I20260620 04:25:39.091778 8634 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2345fcf611134eb7bfe8db60c8aa95a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2345fcf611134eb7bfe8db60c8aa95a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36971 } }
I20260620 04:25:39.091919 8634 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2345fcf611134eb7bfe8db60c8aa95a6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:25:39.091995 8634 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2345fcf611134eb7bfe8db60c8aa95a6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:25:39.092092 8634 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2345fcf611134eb7bfe8db60c8aa95a6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:25:39.092947 8634 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2345fcf611134eb7bfe8db60c8aa95a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2345fcf611134eb7bfe8db60c8aa95a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36971 } }
I20260620 04:25:39.093257 8634 leader_election.cc:302] T 00000000000000000000000000000000 P 2345fcf611134eb7bfe8db60c8aa95a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2345fcf611134eb7bfe8db60c8aa95a6; no voters:
I20260620 04:25:39.093535 8634 leader_election.cc:288] T 00000000000000000000000000000000 P 2345fcf611134eb7bfe8db60c8aa95a6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:25:39.093681 8639 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2345fcf611134eb7bfe8db60c8aa95a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:25:39.093902 8639 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2345fcf611134eb7bfe8db60c8aa95a6 [term 1 LEADER]: Becoming Leader. State: Replica: 2345fcf611134eb7bfe8db60c8aa95a6, State: Running, Role: LEADER
I20260620 04:25:39.094190 8639 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2345fcf611134eb7bfe8db60c8aa95a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2345fcf611134eb7bfe8db60c8aa95a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36971 } }
I20260620 04:25:39.094666 8634 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2345fcf611134eb7bfe8db60c8aa95a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:25:39.095660 8641 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2345fcf611134eb7bfe8db60c8aa95a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2345fcf611134eb7bfe8db60c8aa95a6. Latest consensus state: current_term: 1 leader_uuid: "2345fcf611134eb7bfe8db60c8aa95a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2345fcf611134eb7bfe8db60c8aa95a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36971 } } }
I20260620 04:25:39.095781 8641 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2345fcf611134eb7bfe8db60c8aa95a6 [sys.catalog]: This master's current role is: LEADER
I20260620 04:25:39.096175 8648 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:25:39.096205 8640 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2345fcf611134eb7bfe8db60c8aa95a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2345fcf611134eb7bfe8db60c8aa95a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2345fcf611134eb7bfe8db60c8aa95a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36971 } } }
I20260620 04:25:39.096385 8640 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2345fcf611134eb7bfe8db60c8aa95a6 [sys.catalog]: This master's current role is: LEADER
I20260620 04:25:39.099143 8648 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:25:39.104791 8648 catalog_manager.cc:1347] Generated new cluster ID: e13b27fd1154419982b026c7d3e7ad30
I20260620 04:25:39.104864 8648 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:25:39.120241 8648 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:25:39.121354 8648 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:25:39.126117 8648 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2345fcf611134eb7bfe8db60c8aa95a6: Generated new TSK 0
I20260620 04:25:39.126768 8648 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 04:25:39.155215 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36971
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:25:39.263235 8660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:25:39.263497 8660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:25:39.263541 8660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:25:39.267558 8660 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:25:39.267619 8660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:25:39.267683 8660 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 04:25:39.272868 8660 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36971
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8660
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:25:39.273849 8660 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:25:39.274816 8660 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:25:39.284107 8665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:25:39.284140 8668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:25:39.284147 8660 server_base.cc:1034] running on GCE node
W20260620 04:25:39.284140 8666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:25:39.284822 8660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:25:39.285532 8660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:25:39.286715 8660 hybrid_clock.cc:648] HybridClock initialized: now 1781929539286688 us; error 38 us; skew 500 ppm
Jun 20 04:25:39 dist-test-slave-nk6d krb5kdc[8553](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929539, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:25:39.289423 8660 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 04:25:39.290287 8660 webserver.cc:460] Webserver started at http://127.0.98.1:45131/ using document root <none> and password file <none>
I20260620 04:25:39.290807 8660 fs_manager.cc:359] Metadata directory not provided
I20260620 04:25:39.290867 8660 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:25:39.291065 8660 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:25:39.293063 8660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/ts-0/data/instance:
uuid: "7375394b3c5e4dd7ba544ddadf5a6ecc"
format_stamp: "Formatted at 2026-06-20 04:25:39 on dist-test-slave-nk6d"
server_key: "e804310aab49295742f42f8663ef0be3"
server_key_iv: "4a45a2d8f605fd41f787ec985d953dfd"
server_key_version: "encryptionkey@0"
I20260620 04:25:39.293582 8660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/ts-0/wal/instance:
uuid: "7375394b3c5e4dd7ba544ddadf5a6ecc"
format_stamp: "Formatted at 2026-06-20 04:25:39 on dist-test-slave-nk6d"
server_key: "e804310aab49295742f42f8663ef0be3"
server_key_iv: "4a45a2d8f605fd41f787ec985d953dfd"
server_key_version: "encryptionkey@0"
I20260620 04:25:39.297178 8660 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 04:25:39.299615 8675 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:39.300875 8660 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:25:39.300987 8660 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/ts-0/wal
uuid: "7375394b3c5e4dd7ba544ddadf5a6ecc"
format_stamp: "Formatted at 2026-06-20 04:25:39 on dist-test-slave-nk6d"
server_key: "e804310aab49295742f42f8663ef0be3"
server_key_iv: "4a45a2d8f605fd41f787ec985d953dfd"
server_key_version: "encryptionkey@0"
I20260620 04:25:39.301085 8660 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:25:39.330404 8660 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:25:39.333643 8660 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:25:39.333808 8660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:25:39.334394 8660 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:25:39.335222 8660 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:25:39.335268 8660 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:39.335321 8660 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:25:39.335347 8660 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:39.343314 8660 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37763
I20260620 04:25:39.343348 8788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37763 every 8 connection(s)
I20260620 04:25:39.344192 8660 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/ts-0/data/info.pb
I20260620 04:25:39.351351 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 8660
I20260620 04:25:39.351487 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/ts-0/wal/instance
I20260620 04:25:39.351809 392 external_mini_cluster.cc:1428] Setting key c22e1b208163037d68de05ac49c521c9
Jun 20 04:25:39 dist-test-slave-nk6d krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929539, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:25:39.357658 8637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:39.345904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47541 (local address 127.0.98.62:36971)
0620 04:25:39.346028 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:39.346033 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:39.346879 (+ 846us) server_negotiation.cc:407] Connection header received
0620 04:25:39.348279 (+ 1400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:39.348282 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:39.348330 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:39.348411 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:39.349541 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:39.350067 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:39.351227 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:39.351559 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:39.351670 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:39.354131 (+ 2461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:39.354135 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:39.354138 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:25:39.354165 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:39.355502 (+ 1337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:39.356079 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:39.356082 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:39.356083 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:39.356155 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:39.356547 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:39.356550 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:39.356551 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:39.356729 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:39.356809 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:39.357430 (+ 621us) server_negotiation.cc:299] Negotiation successful
0620 04:25:39.357541 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 04:25:39.358763 8791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:39.346278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36971 (local address 127.0.98.1:47541)
0620 04:25:39.346731 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:39.346783 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:39.348063 (+ 1280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:39.348597 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:39.348603 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:39.348936 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:39.349410 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:39.349422 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:39.350173 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:39.350176 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:39.351108 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:39.351117 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:39.351719 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:39.351722 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:39.351892 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:39.352565 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:39.352581 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:39.353964 (+ 1383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:25:39.355608 (+ 1644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:39.355614 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:39.355630 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:39.355986 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:39.356232 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:39.356235 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:39.356237 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:39.356400 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:39.356832 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:25:39.356839 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:25:39.357168 (+ 329us) client_negotiation.cc:769] Sending connection context
0620 04:25:39.357375 (+ 207us) client_negotiation.cc:240] Negotiation successful
0620 04:25:39.357708 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":111,"threads_started":1}
I20260620 04:25:39.359941 8789 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36971
I20260620 04:25:39.360191 8789 heartbeater.cc:464] Registering TS with master...
I20260620 04:25:39.360641 8789 heartbeater.cc:511] Master 127.0.98.62:36971 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 04:25:39.362046 8597 ts_manager.cc:194] Registered new tserver with Master: 7375394b3c5e4dd7ba544ddadf5a6ecc (127.0.98.1:37763)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 04:25:39.363169 8597 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:47541
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 04:25:39.378371 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36971
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:25:39.484403 8794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:25:39.484639 8794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:25:39.484683 8794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:25:39.488678 8794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:25:39.488734 8794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:25:39.488802 8794 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 04:25:39.494014 8794 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36971
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8794
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:25:39.495081 8794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:25:39.496166 8794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:25:39.505343 8802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:25:39.505373 8799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:25:39.505344 8800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:25:39.505635 8794 server_base.cc:1034] running on GCE node
I20260620 04:25:39.506028 8794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:25:39.506639 8794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:25:39.507797 8794 hybrid_clock.cc:648] HybridClock initialized: now 1781929539507773 us; error 33 us; skew 500 ppm
Jun 20 04:25:39 dist-test-slave-nk6d krb5kdc[8553](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929539, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:25:39.510401 8794 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 04:25:39.511283 8794 webserver.cc:460] Webserver started at http://127.0.98.2:35327/ using document root <none> and password file <none>
I20260620 04:25:39.511802 8794 fs_manager.cc:359] Metadata directory not provided
I20260620 04:25:39.511859 8794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:25:39.512075 8794 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:25:39.514025 8794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/ts-1/data/instance:
uuid: "23692e0198914b369645caf257bb274f"
format_stamp: "Formatted at 2026-06-20 04:25:39 on dist-test-slave-nk6d"
server_key: "7c1c0836a7475d0c2599fb8c2904b964"
server_key_iv: "8e88a9342cc7a5d01e32b20f0aa34a74"
server_key_version: "encryptionkey@0"
I20260620 04:25:39.514544 8794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/ts-1/wal/instance:
uuid: "23692e0198914b369645caf257bb274f"
format_stamp: "Formatted at 2026-06-20 04:25:39 on dist-test-slave-nk6d"
server_key: "7c1c0836a7475d0c2599fb8c2904b964"
server_key_iv: "8e88a9342cc7a5d01e32b20f0aa34a74"
server_key_version: "encryptionkey@0"
I20260620 04:25:39.518064 8794 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260620 04:25:39.520244 8809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:39.521552 8794 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 04:25:39.521654 8794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/ts-1/wal
uuid: "23692e0198914b369645caf257bb274f"
format_stamp: "Formatted at 2026-06-20 04:25:39 on dist-test-slave-nk6d"
server_key: "7c1c0836a7475d0c2599fb8c2904b964"
server_key_iv: "8e88a9342cc7a5d01e32b20f0aa34a74"
server_key_version: "encryptionkey@0"
I20260620 04:25:39.521739 8794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:25:39.533054 8794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:25:39.536125 8794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:25:39.536294 8794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:25:39.536837 8794 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:25:39.537626 8794 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:25:39.537671 8794 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:39.537721 8794 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:25:39.537746 8794 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:39.546802 8794 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37369
I20260620 04:25:39.546834 8922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37369 every 8 connection(s)
I20260620 04:25:39.547865 8794 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/ts-1/data/info.pb
I20260620 04:25:39.553931 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 8794
I20260620 04:25:39.554059 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/ts-1/wal/instance
I20260620 04:25:39.554319 392 external_mini_cluster.cc:1428] Setting key 5636221c8d6d77260fb3d1a6032e934e
Jun 20 04:25:39 dist-test-slave-nk6d krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929539, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:25:39.561422 8637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:39.549649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33959 (local address 127.0.98.62:36971)
0620 04:25:39.549770 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:39.549774 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:39.550609 (+ 835us) server_negotiation.cc:407] Connection header received
0620 04:25:39.551930 (+ 1321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:39.551933 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:39.552038 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:39.552111 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:39.553276 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:39.553858 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:39.555244 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:39.555458 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:39.555544 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:39.558027 (+ 2483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:39.558031 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:39.558034 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:25:39.558062 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:39.559342 (+ 1280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:39.559903 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:39.559906 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:39.559907 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:39.560013 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:39.560366 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:39.560368 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:39.560369 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:39.560539 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:39.560614 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:39.561211 (+ 597us) server_negotiation.cc:299] Negotiation successful
0620 04:25:39.561305 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 04:25:39.562593 8925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:39.550020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36971 (local address 127.0.98.2:33959)
0620 04:25:39.550446 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:39.550490 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:39.551726 (+ 1236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:39.552295 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:39.552301 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:39.552649 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:39.553132 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:39.553146 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:39.554008 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:39.554012 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:39.555131 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:39.555142 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:39.555596 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:39.555600 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:39.555767 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:39.556338 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:39.556354 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:39.557862 (+ 1508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:25:39.559448 (+ 1586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:39.559454 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:39.559471 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:39.559812 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:39.560098 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:39.560101 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:39.560103 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:39.560284 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:39.560635 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:25:39.560641 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:25:39.560966 (+ 325us) client_negotiation.cc:769] Sending connection context
0620 04:25:39.561189 (+ 223us) client_negotiation.cc:240] Negotiation successful
0620 04:25:39.561538 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":97,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 04:25:39.563719 8923 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36971
I20260620 04:25:39.563937 8923 heartbeater.cc:464] Registering TS with master...
I20260620 04:25:39.564438 8923 heartbeater.cc:511] Master 127.0.98.62:36971 requested a full tablet report, sending...
I20260620 04:25:39.565454 8597 ts_manager.cc:194] Registered new tserver with Master: 23692e0198914b369645caf257bb274f (127.0.98.2:37369)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260620 04:25:39.566112 8597 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:33959
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 04:25:39.578763 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36971
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:25:39.683626 8928 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:25:39.683859 8928 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:25:39.683904 8928 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:25:39.687866 8928 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:25:39.687925 8928 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:25:39.688019 8928 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 04:25:39.693114 8928 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36971
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8928
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:25:39.694059 8928 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:25:39.695027 8928 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 04:25:39.704071 8928 server_base.cc:1034] running on GCE node
W20260620 04:25:39.703999 8934 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:25:39.704010 8933 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:25:39.704010 8936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:25:39.704779 8928 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:25:39.705338 8928 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:25:39.706477 8928 hybrid_clock.cc:648] HybridClock initialized: now 1781929539706452 us; error 38 us; skew 500 ppm
Jun 20 04:25:39 dist-test-slave-nk6d krb5kdc[8553](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929539, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:25:39.708952 8928 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 04:25:39.709753 8928 webserver.cc:460] Webserver started at http://127.0.98.3:35955/ using document root <none> and password file <none>
I20260620 04:25:39.710251 8928 fs_manager.cc:359] Metadata directory not provided
I20260620 04:25:39.710312 8928 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:25:39.710482 8928 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:25:39.712734 8928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/ts-2/data/instance:
uuid: "6ad8a96da8d7443ebce978fe8f16ef15"
format_stamp: "Formatted at 2026-06-20 04:25:39 on dist-test-slave-nk6d"
server_key: "6f9f2d8dd0302faa219d9aa4822b0bba"
server_key_iv: "ee4401ee11a817a44a86f59af77b1791"
server_key_version: "encryptionkey@0"
I20260620 04:25:39.713308 8928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/ts-2/wal/instance:
uuid: "6ad8a96da8d7443ebce978fe8f16ef15"
format_stamp: "Formatted at 2026-06-20 04:25:39 on dist-test-slave-nk6d"
server_key: "6f9f2d8dd0302faa219d9aa4822b0bba"
server_key_iv: "ee4401ee11a817a44a86f59af77b1791"
server_key_version: "encryptionkey@0"
I20260620 04:25:39.716806 8928 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:25:39.718986 8943 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:39.720155 8928 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:25:39.720252 8928 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/ts-2/wal
uuid: "6ad8a96da8d7443ebce978fe8f16ef15"
format_stamp: "Formatted at 2026-06-20 04:25:39 on dist-test-slave-nk6d"
server_key: "6f9f2d8dd0302faa219d9aa4822b0bba"
server_key_iv: "ee4401ee11a817a44a86f59af77b1791"
server_key_version: "encryptionkey@0"
I20260620 04:25:39.720340 8928 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:25:39.738489 8928 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:25:39.741287 8928 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:25:39.741459 8928 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:25:39.742017 8928 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:25:39.742802 8928 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:25:39.742847 8928 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:39.742898 8928 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:25:39.742926 8928 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:39.750978 8928 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35463
I20260620 04:25:39.750999 9056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35463 every 8 connection(s)
I20260620 04:25:39.751807 8928 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/ts-2/data/info.pb
I20260620 04:25:39.753866 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 8928
I20260620 04:25:39.753957 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/ts-2/wal/instance
I20260620 04:25:39.754298 392 external_mini_cluster.cc:1428] Setting key 45b507a7fa1a05800bb7b08ea8012190
Jun 20 04:25:39 dist-test-slave-nk6d krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929539, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:25:39.763902 8637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:39.753450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59797 (local address 127.0.98.62:36971)
0620 04:25:39.753570 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:39.753592 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:39.754460 (+ 868us) server_negotiation.cc:407] Connection header received
0620 04:25:39.756036 (+ 1576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:39.756040 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:39.756083 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:39.756158 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:39.757279 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:39.757783 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:39.758823 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:39.759038 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:39.759105 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:39.761265 (+ 2160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:39.761269 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:39.761271 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:25:39.761293 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:39.762372 (+ 1079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:39.762793 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:39.762795 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:39.762796 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:39.762859 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:39.763088 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:39.763090 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:39.763091 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:39.763240 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:39.763307 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:39.763702 (+ 395us) server_negotiation.cc:299] Negotiation successful
0620 04:25:39.763790 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 04:25:39.764688 9059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:39.753799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36971 (local address 127.0.98.3:59797)
0620 04:25:39.754303 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:39.754358 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:39.755793 (+ 1435us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:39.756345 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:39.756354 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:39.756696 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:39.757161 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:39.757173 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:39.757872 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:39.757874 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:39.758737 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:39.758744 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:39.759136 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:39.759139 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:39.759273 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:39.759795 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:39.759810 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:39.761130 (+ 1320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:25:39.762458 (+ 1328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:39.762463 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:39.762474 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:39.762723 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:39.762916 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:39.762918 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:39.762919 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:39.763028 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:39.763306 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:25:39.763310 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:25:39.763510 (+ 200us) client_negotiation.cc:769] Sending connection context
0620 04:25:39.763670 (+ 160us) client_negotiation.cc:240] Negotiation successful
0620 04:25:39.763889 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":149,"threads_started":1}
I20260620 04:25:39.765625 9057 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36971
I20260620 04:25:39.765818 9057 heartbeater.cc:464] Registering TS with master...
I20260620 04:25:39.766223 9057 heartbeater.cc:511] Master 127.0.98.62:36971 requested a full tablet report, sending...
I20260620 04:25:39.767181 8598 ts_manager.cc:194] Registered new tserver with Master: 6ad8a96da8d7443ebce978fe8f16ef15 (127.0.98.3:35463)
I20260620 04:25:39.767838 8598 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59797
I20260620 04:25:39.767863 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260620 04:25:39.830382 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 8660
I20260620 04:25:39.835211 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 8794
I20260620 04:25:39.839705 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 8928
I20260620 04:25:39.844018 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 8566
I20260620 04:25:39.848788 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36971
--webserver_interface=127.0.98.62
--webserver_port=42025
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36971
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:25:39.955855 9066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:25:39.956154 9066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:25:39.956194 9066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:25:39.960187 9066 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:25:39.960249 9066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:25:39.960268 9066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:25:39.960284 9066 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:25:39.960299 9066 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:25:39.965641 9066 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36971
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36971
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=42025
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9066
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:25:39.966710 9066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:25:39.967777 9066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:25:39.976418 9071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:25:39.976438 9072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:25:39.976454 9074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:25:39.976717 9066 server_base.cc:1034] running on GCE node
I20260620 04:25:39.977270 9066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:25:39.978333 9066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:25:39.979488 9066 hybrid_clock.cc:648] HybridClock initialized: now 1781929539979473 us; error 32 us; skew 500 ppm
Jun 20 04:25:39 dist-test-slave-nk6d krb5kdc[8553](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929539, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:25:39.982228 9066 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 04:25:39.983204 9066 webserver.cc:460] Webserver started at http://127.0.98.62:42025/ using document root <none> and password file <none>
I20260620 04:25:39.983744 9066 fs_manager.cc:359] Metadata directory not provided
I20260620 04:25:39.983816 9066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:25:39.987715 9066 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:25:39.989871 9081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:39.991074 9066 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:25:39.991185 9066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/master-0/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/master-0/wal
uuid: "2345fcf611134eb7bfe8db60c8aa95a6"
format_stamp: "Formatted at 2026-06-20 04:25:39 on dist-test-slave-nk6d"
server_key: "994ed8c2bb27006c49af45cb4f281b12"
server_key_iv: "ee254f859d171fb5383d6b2c0ff4a39b"
server_key_version: "encryptionkey@0"
I20260620 04:25:39.991729 9066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:25:40.003551 9066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:25:40.006698 9066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:25:40.006877 9066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:25:40.013656 9066 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36971
I20260620 04:25:40.013716 9133 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36971 every 8 connection(s)
I20260620 04:25:40.014716 9066 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:25:40.018813 9134 sys_catalog.cc:263] Verifying existing consensus state
I20260620 04:25:40.020447 9134 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2345fcf611134eb7bfe8db60c8aa95a6: Bootstrap starting.
I20260620 04:25:40.024236 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 9066
I20260620 04:25:40.024688 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:37763
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:36971
--webserver_port=45131
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 04:25:40.036584 9134 log.cc:824] T 00000000000000000000000000000000 P 2345fcf611134eb7bfe8db60c8aa95a6: Log is configured to *not* fsync() on all Append() calls
I20260620 04:25:40.043058 9134 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2345fcf611134eb7bfe8db60c8aa95a6: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 04:25:40.043385 9134 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2345fcf611134eb7bfe8db60c8aa95a6: Bootstrap complete.
I20260620 04:25:40.047441 9134 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2345fcf611134eb7bfe8db60c8aa95a6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2345fcf611134eb7bfe8db60c8aa95a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36971 } }
I20260620 04:25:40.047839 9134 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2345fcf611134eb7bfe8db60c8aa95a6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2345fcf611134eb7bfe8db60c8aa95a6, State: Initialized, Role: FOLLOWER
I20260620 04:25:40.048348 9134 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2345fcf611134eb7bfe8db60c8aa95a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2345fcf611134eb7bfe8db60c8aa95a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36971 } }
I20260620 04:25:40.048458 9134 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2345fcf611134eb7bfe8db60c8aa95a6 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:25:40.048518 9134 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2345fcf611134eb7bfe8db60c8aa95a6 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:25:40.048615 9134 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2345fcf611134eb7bfe8db60c8aa95a6 [term 1 FOLLOWER]: Advancing to term 2
I20260620 04:25:40.050482 9134 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2345fcf611134eb7bfe8db60c8aa95a6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2345fcf611134eb7bfe8db60c8aa95a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36971 } }
I20260620 04:25:40.050817 9134 leader_election.cc:302] T 00000000000000000000000000000000 P 2345fcf611134eb7bfe8db60c8aa95a6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2345fcf611134eb7bfe8db60c8aa95a6; no voters:
I20260620 04:25:40.051110 9134 leader_election.cc:288] T 00000000000000000000000000000000 P 2345fcf611134eb7bfe8db60c8aa95a6 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 04:25:40.051210 9138 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2345fcf611134eb7bfe8db60c8aa95a6 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 04:25:40.051393 9138 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2345fcf611134eb7bfe8db60c8aa95a6 [term 2 LEADER]: Becoming Leader. State: Replica: 2345fcf611134eb7bfe8db60c8aa95a6, State: Running, Role: LEADER
I20260620 04:25:40.051678 9138 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2345fcf611134eb7bfe8db60c8aa95a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2345fcf611134eb7bfe8db60c8aa95a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36971 } }
I20260620 04:25:40.052342 9134 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2345fcf611134eb7bfe8db60c8aa95a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:25:40.052964 9139 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2345fcf611134eb7bfe8db60c8aa95a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "2345fcf611134eb7bfe8db60c8aa95a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2345fcf611134eb7bfe8db60c8aa95a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36971 } } }
I20260620 04:25:40.053104 9139 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2345fcf611134eb7bfe8db60c8aa95a6 [sys.catalog]: This master's current role is: LEADER
I20260620 04:25:40.053824 9140 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2345fcf611134eb7bfe8db60c8aa95a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2345fcf611134eb7bfe8db60c8aa95a6. Latest consensus state: current_term: 2 leader_uuid: "2345fcf611134eb7bfe8db60c8aa95a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2345fcf611134eb7bfe8db60c8aa95a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36971 } } }
I20260620 04:25:40.053982 9140 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2345fcf611134eb7bfe8db60c8aa95a6 [sys.catalog]: This master's current role is: LEADER
I20260620 04:25:40.057456 9154 catalog_manager.cc:1259] Loaded cluster ID: e13b27fd1154419982b026c7d3e7ad30
I20260620 04:25:40.057518 9154 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2345fcf611134eb7bfe8db60c8aa95a6: loading cluster ID for follower catalog manager: success
I20260620 04:25:40.059875 9154 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2345fcf611134eb7bfe8db60c8aa95a6: acquiring CA information for follower catalog manager: success
I20260620 04:25:40.061240 9154 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2345fcf611134eb7bfe8db60c8aa95a6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 04:25:40.061376 9146 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:25:40.062220 9146 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:25:40.062615 9146 catalog_manager.cc:1259] Loaded cluster ID: e13b27fd1154419982b026c7d3e7ad30
I20260620 04:25:40.062667 9146 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:25:40.063259 9146 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:25:40.063705 9146 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2345fcf611134eb7bfe8db60c8aa95a6: Loaded TSK: 0
I20260620 04:25:40.064471 9146 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 04:25:40.138031 9136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:25:40.138291 9136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:25:40.138334 9136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:25:40.142377 9136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:25:40.142437 9136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:25:40.142498 9136 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 04:25:40.147886 9136 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:37763
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=45131
--tserver_master_addrs=127.0.98.62:36971
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9136
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:25:40.148977 9136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:25:40.149979 9136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:25:40.159408 9161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:25:40.159441 9160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:25:40.159416 9163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:25:40.160084 9136 server_base.cc:1034] running on GCE node
I20260620 04:25:40.160566 9136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:25:40.161379 9136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:25:40.162575 9136 hybrid_clock.cc:648] HybridClock initialized: now 1781929540162541 us; error 44 us; skew 500 ppm
Jun 20 04:25:40 dist-test-slave-nk6d krb5kdc[8553](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929540, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:25:40.165886 9136 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260620 04:25:40.166985 9136 webserver.cc:460] Webserver started at http://127.0.98.1:45131/ using document root <none> and password file <none>
I20260620 04:25:40.167660 9136 fs_manager.cc:359] Metadata directory not provided
I20260620 04:25:40.167758 9136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:25:40.172333 9136 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 04:25:40.174714 9170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:40.176316 9136 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:25:40.176424 9136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/ts-0/wal
uuid: "7375394b3c5e4dd7ba544ddadf5a6ecc"
format_stamp: "Formatted at 2026-06-20 04:25:39 on dist-test-slave-nk6d"
server_key: "e804310aab49295742f42f8663ef0be3"
server_key_iv: "4a45a2d8f605fd41f787ec985d953dfd"
server_key_version: "encryptionkey@0"
I20260620 04:25:40.176964 9136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:25:40.199676 9136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:25:40.202591 9136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:25:40.202772 9136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:25:40.203344 9136 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:25:40.204190 9136 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:25:40.204238 9136 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:40.204290 9136 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:25:40.204317 9136 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:40.212692 9136 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37763
I20260620 04:25:40.212730 9283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37763 every 8 connection(s)
I20260620 04:25:40.213629 9136 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/ts-0/data/info.pb
I20260620 04:25:40.220808 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 9136
I20260620 04:25:40.221199 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:37369
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:36971
--webserver_port=35327
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 04:25:40 dist-test-slave-nk6d krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:25:40 dist-test-slave-nk6d krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:25:40.226137 9287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:40.215713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36971 (local address 127.0.98.1:49023)
0620 04:25:40.216173 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:40.216222 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:40.217419 (+ 1197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:40.218396 (+ 977us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:40.218402 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:40.218767 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:40.219216 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:40.219229 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:40.220427 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:40.220430 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:40.221345 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:40.221355 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:40.222299 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:40.222302 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:40.222482 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:40.223063 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:40.223083 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:40.225403 (+ 2320us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36971: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":116,"threads_started":1}
W20260620 04:25:40.226603 9286 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:25:40.215652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49023 (local address 127.0.98.62:36971)
0620 04:25:40.216076 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:40.216094 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:40.216320 (+ 226us) server_negotiation.cc:407] Connection header received
0620 04:25:40.217770 (+ 1450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:40.217782 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:40.218061 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:40.218330 (+ 269us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:40.219358 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:40.220326 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:40.221478 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:40.222161 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:40.222312 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:40.225687 (+ 3375us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:49023: BlockingRecv error: recv got EOF from 127.0.98.1:49023 (error 108)
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":99,"threads_started":1}
W20260620 04:25:40.226671 9284 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36971 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:36971: Client connection negotiation failed: client connection to 127.0.98.62:36971: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Jun 20 04:25:40 dist-test-slave-nk6d krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:25:40 dist-test-slave-nk6d krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:25:40.231648 9287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:40.227061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36971 (local address 127.0.98.1:36835)
0620 04:25:40.227217 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:40.227228 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:40.227297 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:40.227515 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:40.227518 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:40.227666 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:40.227749 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:40.227755 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:40.228535 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:40.228539 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:40.229069 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:40.229076 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:40.229483 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:40.229486 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:40.229569 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:40.229984 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:40.229996 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:40.231512 (+ 1516us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36971: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":94}
W20260620 04:25:40.231746 9286 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:25:40.227218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36835 (local address 127.0.98.62:36971)
0620 04:25:40.227332 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:40.227336 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:40.227348 (+ 12us) server_negotiation.cc:407] Connection header received
0620 04:25:40.227383 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:40.227386 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:40.227428 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:40.227491 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:40.227852 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:40.228439 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:40.229175 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:40.229393 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:40.229476 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:40.231648 (+ 2172us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:36835: BlockingRecv error: recv got EOF from 127.0.98.1:36835 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Jun 20 04:25:40 dist-test-slave-nk6d krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:25:40 dist-test-slave-nk6d krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:25:40.236773 9287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:40.232151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36971 (local address 127.0.98.1:51757)
0620 04:25:40.232441 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:40.232452 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:40.232530 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:40.232735 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:40.232737 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:40.232873 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:40.232940 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:40.232945 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:40.233618 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:40.233621 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:40.234129 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:40.234137 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:40.234515 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:40.234518 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:40.234588 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:40.235007 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:40.235021 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:40.236625 (+ 1604us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36971: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":93,"threads_started":1}
W20260620 04:25:40.236826 9286 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:25:40.232233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51757 (local address 127.0.98.62:36971)
0620 04:25:40.232357 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:40.232361 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:40.232510 (+ 149us) server_negotiation.cc:407] Connection header received
0620 04:25:40.232602 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:40.232604 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:40.232649 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:40.232710 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:40.233036 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:40.233524 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:40.234227 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:40.234429 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:40.234509 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:40.236737 (+ 2228us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:51757: BlockingRecv error: recv got EOF from 127.0.98.1:51757 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260620 04:25:40.237056 9284 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 04:25:40.329918 9288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:25:40.330173 9288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:25:40.330215 9288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:25:40.334110 9288 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:25:40.334168 9288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:25:40.334225 9288 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 04:25:40.339442 9288 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:37369
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=35327
--tserver_master_addrs=127.0.98.62:36971
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9288
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:25:40.340483 9288 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:25:40.341485 9288 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:25:40.350745 9294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:25:40.350816 9288 server_base.cc:1034] running on GCE node
W20260620 04:25:40.350754 9295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:25:40.351102 9297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:25:40.351475 9288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:25:40.352114 9288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:25:40.353281 9288 hybrid_clock.cc:648] HybridClock initialized: now 1781929540353260 us; error 33 us; skew 500 ppm
Jun 20 04:25:40 dist-test-slave-nk6d krb5kdc[8553](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929540, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:25:40.356292 9288 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260620 04:25:40.357276 9288 webserver.cc:460] Webserver started at http://127.0.98.2:35327/ using document root <none> and password file <none>
I20260620 04:25:40.357818 9288 fs_manager.cc:359] Metadata directory not provided
I20260620 04:25:40.357896 9288 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:25:40.361794 9288 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 04:25:40.364037 9304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:40.365279 9288 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:25:40.365396 9288 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/ts-1/wal
uuid: "23692e0198914b369645caf257bb274f"
format_stamp: "Formatted at 2026-06-20 04:25:39 on dist-test-slave-nk6d"
server_key: "7c1c0836a7475d0c2599fb8c2904b964"
server_key_iv: "8e88a9342cc7a5d01e32b20f0aa34a74"
server_key_version: "encryptionkey@0"
I20260620 04:25:40.365916 9288 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:25:40.395327 9288 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:25:40.398177 9288 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:25:40.398345 9288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:25:40.398903 9288 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:25:40.399691 9288 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:25:40.399735 9288 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:40.399777 9288 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:25:40.399806 9288 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:40.407980 9288 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37369
I20260620 04:25:40.408021 9417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37369 every 8 connection(s)
I20260620 04:25:40.408989 9288 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/ts-1/data/info.pb
Jun 20 04:25:40 dist-test-slave-nk6d krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:25:40.417933 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 9288
Jun 20 04:25:40 dist-test-slave-nk6d krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:25:40.418361 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:35463
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:36971
--webserver_port=35955
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:25:40.419591 9286 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:25:40.410792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55043 (local address 127.0.98.62:36971)
0620 04:25:40.410966 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:40.410970 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:40.411729 (+ 759us) server_negotiation.cc:407] Connection header received
0620 04:25:40.413029 (+ 1300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:40.413032 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:40.413083 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:40.413170 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:40.414275 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:40.414800 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:40.415884 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:40.416142 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:40.416234 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:40.419493 (+ 3259us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:55043: BlockingRecv error: recv got EOF from 127.0.98.2:55043 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
I20260620 04:25:40.420296 9420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:40.411140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36971 (local address 127.0.98.2:55043)
0620 04:25:40.411574 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:40.411618 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:40.412831 (+ 1213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:40.413318 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:40.413324 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:40.413664 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:40.414144 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:40.414157 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:40.414910 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:40.414912 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:40.415791 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:40.415799 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:40.416268 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:40.416271 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:40.416402 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:40.416888 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:40.416904 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:40.419380 (+ 2476us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36971: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":118,"threads_started":1}
W20260620 04:25:40.420842 9418 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36971 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:36971: Client connection negotiation failed: client connection to 127.0.98.62:36971: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Jun 20 04:25:40 dist-test-slave-nk6d krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:25:40 dist-test-slave-nk6d krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:25:40.426251 9420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:40.421244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36971 (local address 127.0.98.2:54505)
0620 04:25:40.421350 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:40.421364 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:40.421438 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:40.421660 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:40.421663 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:40.421804 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:40.421911 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:40.421918 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:40.422928 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:40.422930 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:40.423480 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:40.423494 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:40.424052 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:40.424056 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:40.424131 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:40.424553 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:40.424567 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:40.426158 (+ 1591us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36971: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":40}
W20260620 04:25:40.426434 9286 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:25:40.421333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54505 (local address 127.0.98.62:36971)
0620 04:25:40.421436 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:40.421440 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:40.421453 (+ 13us) server_negotiation.cc:407] Connection header received
0620 04:25:40.421533 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:40.421536 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:40.421575 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:40.421637 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:40.422036 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:40.422802 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:40.423592 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:40.423922 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:40.424048 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:40.426317 (+ 2269us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:54505: BlockingRecv error: recv got EOF from 127.0.98.2:54505 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Jun 20 04:25:40 dist-test-slave-nk6d krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:25:40 dist-test-slave-nk6d krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:25:40.431221 9420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:40.426885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36971 (local address 127.0.98.2:34221)
0620 04:25:40.427031 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:40.427044 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:40.427121 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:40.427316 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:40.427318 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:40.427452 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:40.427514 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:40.427518 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:40.428243 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:40.428247 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:40.428756 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:40.428763 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:40.429146 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:40.429150 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:40.429221 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:40.429655 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:40.429665 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:40.431096 (+ 1431us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36971: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":86}
W20260620 04:25:40.431391 9418 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 04:25:40.431306 9286 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:25:40.426971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34221 (local address 127.0.98.62:36971)
0620 04:25:40.427079 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:40.427082 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:40.427095 (+ 13us) server_negotiation.cc:407] Connection header received
0620 04:25:40.427194 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:40.427197 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:40.427234 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:40.427294 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:40.427615 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:40.428139 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:40.428851 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:40.429053 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:40.429135 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:40.431212 (+ 2077us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:34221: BlockingRecv error: recv got EOF from 127.0.98.2:34221 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260620 04:25:40.526899 9421 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:25:40.527164 9421 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:25:40.527232 9421 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:25:40.531193 9421 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:25:40.531255 9421 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:25:40.531337 9421 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 04:25:40.536512 9421 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:35463
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=35955
--tserver_master_addrs=127.0.98.62:36971
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9421
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:25:40.537518 9421 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:25:40.538511 9421 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:25:40.547783 9429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:25:40.547861 9426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:25:40.547827 9427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:25:40.548134 9421 server_base.cc:1034] running on GCE node
I20260620 04:25:40.548535 9421 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:25:40.549180 9421 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:25:40.550343 9421 hybrid_clock.cc:648] HybridClock initialized: now 1781929540550320 us; error 35 us; skew 500 ppm
Jun 20 04:25:40 dist-test-slave-nk6d krb5kdc[8553](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929540, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:25:40.553077 9421 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260620 04:25:40.554262 9421 webserver.cc:460] Webserver started at http://127.0.98.3:35955/ using document root <none> and password file <none>
I20260620 04:25:40.554867 9421 fs_manager.cc:359] Metadata directory not provided
I20260620 04:25:40.554939 9421 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:25:40.558605 9421 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:25:40.560649 9436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:40.561885 9421 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:25:40.561980 9421 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/ts-2/wal
uuid: "6ad8a96da8d7443ebce978fe8f16ef15"
format_stamp: "Formatted at 2026-06-20 04:25:39 on dist-test-slave-nk6d"
server_key: "6f9f2d8dd0302faa219d9aa4822b0bba"
server_key_iv: "ee4401ee11a817a44a86f59af77b1791"
server_key_version: "encryptionkey@0"
I20260620 04:25:40.562493 9421 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:25:40.584363 9421 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:25:40.587580 9421 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:25:40.587785 9421 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:25:40.588433 9421 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:25:40.589267 9421 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:25:40.589313 9421 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:40.589355 9421 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:25:40.589385 9421 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:40.598218 9421 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35463
I20260620 04:25:40.598239 9549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35463 every 8 connection(s)
I20260620 04:25:40.599346 9421 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929496330398-392-0/minicluster-data/ts-2/data/info.pb
I20260620 04:25:40.604848 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 9421
Jun 20 04:25:40 dist-test-slave-nk6d krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:25:40 dist-test-slave-nk6d krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 04:25:40.611374 9286 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:25:40.601237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55985 (local address 127.0.98.62:36971)
0620 04:25:40.601362 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:40.601366 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:40.602260 (+ 894us) server_negotiation.cc:407] Connection header received
0620 04:25:40.603592 (+ 1332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:40.603596 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:40.603654 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:40.603729 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:40.604969 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:40.605576 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:40.607078 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:40.607450 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:40.607548 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:40.611235 (+ 3687us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:55985: BlockingRecv error: recv got EOF from 127.0.98.3:55985 (error 108)
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":46}
I20260620 04:25:40.612005 9552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:40.601622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36971 (local address 127.0.98.3:55985)
0620 04:25:40.602103 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:40.602146 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:40.603373 (+ 1227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:40.603920 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:40.603927 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:40.604316 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:40.604822 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:40.604836 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:40.605713 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:40.605717 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:40.606980 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:40.606990 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:40.607588 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:40.607591 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:40.607734 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:40.608318 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:40.608339 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:40.611101 (+ 2762us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36971: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":101,"threads_started":1}
W20260620 04:25:40.612506 9550 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36971 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:36971: Client connection negotiation failed: client connection to 127.0.98.62:36971: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
I20260620 04:25:40.614549 9554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:40.605822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53124 (local address 127.0.98.62:36971)
0620 04:25:40.606024 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:40.606027 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:40.606038 (+ 11us) server_negotiation.cc:407] Connection header received
0620 04:25:40.606066 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:40.606068 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:40.606101 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:40.606169 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:40.607023 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:40.607732 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:40.608852 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:40.609074 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:40.609152 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:40.610582 (+ 1430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:40.610588 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:40.610603 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:25:40.610630 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:40.612419 (+ 1789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:40.612918 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:40.612924 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:40.612931 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:40.613080 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:40.613333 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:40.613336 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:40.613338 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:40.613787 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:40.613929 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:40.614187 (+ 258us) server_negotiation.cc:299] Negotiation successful
0620 04:25:40.614404 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":6,"server-negotiator.queue_time_us":146,"thread_start_us":71,"threads_started":1}
Jun 20 04:25:40 dist-test-slave-nk6d krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:25:40 dist-test-slave-nk6d krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:25:40.617777 9552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:40.612940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36971 (local address 127.0.98.3:45007)
0620 04:25:40.613042 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:40.613055 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:40.613125 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:40.613338 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:40.613340 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:40.613482 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:40.613589 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:40.613596 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:40.614467 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:40.614471 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:40.615117 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:40.615125 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:40.615529 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:40.615533 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:40.615618 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:40.616029 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:40.616045 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:40.617688 (+ 1643us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36971: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":31}
W20260620 04:25:40.617898 9286 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:25:40.613026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45007 (local address 127.0.98.62:36971)
0620 04:25:40.613124 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:40.613128 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:40.613142 (+ 14us) server_negotiation.cc:407] Connection header received
0620 04:25:40.613205 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:40.613208 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:40.613249 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:40.613320 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:40.613689 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:40.614369 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:40.615229 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:40.615440 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:40.615520 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:40.617801 (+ 2281us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:45007: BlockingRecv error: recv got EOF from 127.0.98.3:45007 (error 108)
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":30}
Jun 20 04:25:40 dist-test-slave-nk6d krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:25:40 dist-test-slave-nk6d krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:25:40.622495 9552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:40.618237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36971 (local address 127.0.98.3:57159)
0620 04:25:40.618349 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:40.618363 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:40.618432 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:40.618593 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:40.618595 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:40.618713 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:40.618780 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:40.618784 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:40.619398 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:40.619400 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:40.619907 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:40.619913 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:40.620302 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:40.620304 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:40.620366 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:40.620804 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:40.620817 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:40.622376 (+ 1559us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36971: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":40}
W20260620 04:25:40.622717 9286 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:25:40.618278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57159 (local address 127.0.98.62:36971)
0620 04:25:40.618412 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:40.618415 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:40.618426 (+ 11us) server_negotiation.cc:407] Connection header received
0620 04:25:40.618486 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:40.618488 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:40.618520 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:40.618576 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:40.618857 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:40.619323 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:40.620010 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:40.620212 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:40.620279 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:40.622513 (+ 2234us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:57159: BlockingRecv error: recv got EOF from 127.0.98.3:57159 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260620 04:25:40.622795 9550 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 20 04:25:41 dist-test-slave-nk6d krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:25:41 dist-test-slave-nk6d krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:25:41.246889 9555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:41.237663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36971 (local address 127.0.98.1:42607)
0620 04:25:41.237938 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:41.237951 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:41.238036 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:41.238250 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:41.238253 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:41.238434 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:41.238549 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:41.238555 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:41.239298 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:41.239301 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:41.239880 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:41.239888 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:41.240369 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:41.240372 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:41.240446 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:41.244974 (+ 4528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:41.244988 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:41.246743 (+ 1755us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36971: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":79,"threads_started":1}
W20260620 04:25:41.247016 9556 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:25:41.237781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42607 (local address 127.0.98.62:36971)
0620 04:25:41.237996 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:41.238000 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:41.238019 (+ 19us) server_negotiation.cc:407] Connection header received
0620 04:25:41.238103 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:41.238105 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:41.238147 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:41.238230 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:41.238636 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:41.239187 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:41.240026 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:41.240261 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:41.240342 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:41.246887 (+ 6545us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:42607: BlockingRecv error: recv got EOF from 127.0.98.1:42607 (error 108)
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":64,"threads_started":1}
Jun 20 04:25:41 dist-test-slave-nk6d krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:25:41 dist-test-slave-nk6d krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:25:41.437232 9557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:41.432070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36971 (local address 127.0.98.2:44009)
0620 04:25:41.432349 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:41.432361 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:41.432468 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:41.432706 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:41.432708 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:41.432905 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:41.433011 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:41.433017 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:41.433779 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:41.433783 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:41.434352 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:41.434360 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:41.434777 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:41.434780 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:41.434881 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:41.435347 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:41.435358 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:41.437065 (+ 1707us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36971: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":83,"threads_started":1}
W20260620 04:25:41.437319 9556 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:25:41.432111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44009 (local address 127.0.98.62:36971)
0620 04:25:41.432245 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:41.432249 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:41.432429 (+ 180us) server_negotiation.cc:407] Connection header received
0620 04:25:41.432558 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:41.432561 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:41.432604 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:41.432670 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:41.433121 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:41.433642 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:41.434463 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:41.434676 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:41.434759 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:41.437211 (+ 2452us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:44009: BlockingRecv error: recv got EOF from 127.0.98.2:44009 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Jun 20 04:25:41 dist-test-slave-nk6d krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:25:41 dist-test-slave-nk6d krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:25:41.628602 9558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:41.623451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36971 (local address 127.0.98.3:49503)
0620 04:25:41.623700 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:41.623713 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:41.623813 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:41.624052 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:41.624054 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:41.624226 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:41.624341 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:41.624346 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:41.625113 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:41.625117 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:41.625689 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:41.625698 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:41.626058 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:41.626060 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:41.626125 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:41.626572 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:41.626583 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:41.628462 (+ 1879us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36971: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":68,"threads_started":1}
W20260620 04:25:41.628708 9556 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:25:41.623530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49503 (local address 127.0.98.62:36971)
0620 04:25:41.623664 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:41.623667 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:41.623787 (+ 120us) server_negotiation.cc:407] Connection header received
0620 04:25:41.623881 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:41.623883 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:41.623930 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:41.624015 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:41.624426 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:41.624994 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:41.625785 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:41.625985 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:41.626047 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:41.628594 (+ 2547us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:49503: BlockingRecv error: recv got EOF from 127.0.98.3:49503 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 20 04:25:42 dist-test-slave-nk6d krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:25:42 dist-test-slave-nk6d krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:25:42.254292 9559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:42.247627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36971 (local address 127.0.98.1:44419)
0620 04:25:42.247912 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:42.247928 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:42.248082 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:42.248369 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:42.248372 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:42.248597 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:42.248720 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:42.248729 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:42.249790 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:42.249794 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:42.250625 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:42.250636 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:42.251161 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:42.251164 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:42.251252 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:42.251829 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:42.251845 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:42.254141 (+ 2296us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36971: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":109,"threads_started":1}
W20260620 04:25:42.254395 9560 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:25:42.247699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44419 (local address 127.0.98.62:36971)
0620 04:25:42.248082 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:42.248086 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:42.248100 (+ 14us) server_negotiation.cc:407] Connection header received
0620 04:25:42.248181 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:42.248184 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:42.248232 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:42.248312 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:42.248854 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:42.249675 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:42.250734 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:42.251037 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:42.251138 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:42.254280 (+ 3142us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:44419: BlockingRecv error: recv got EOF from 127.0.98.1:44419 (error 108)
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":72,"threads_started":1}
Jun 20 04:25:42 dist-test-slave-nk6d krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:25:42 dist-test-slave-nk6d krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:25:42.443167 9561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:42.438043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36971 (local address 127.0.98.2:38773)
0620 04:25:42.438303 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:42.438317 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:42.438416 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:42.438709 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:42.438711 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:42.438882 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:42.438978 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:42.438983 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:42.439676 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:42.439679 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:42.440289 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:42.440299 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:42.440731 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:42.440733 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:42.440802 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:42.441289 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:42.441301 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:42.443026 (+ 1725us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36971: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":68,"threads_started":1}
W20260620 04:25:42.443272 9560 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:25:42.438171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38773 (local address 127.0.98.62:36971)
0620 04:25:42.438339 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:42.438342 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:42.438362 (+ 20us) server_negotiation.cc:407] Connection header received
0620 04:25:42.438530 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:42.438532 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:42.438580 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:42.438649 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:42.439064 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:42.439578 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:42.440397 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:42.440625 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:42.440709 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:42.443158 (+ 2449us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:38773: BlockingRecv error: recv got EOF from 127.0.98.2:38773 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Jun 20 04:25:42 dist-test-slave-nk6d krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:25:42 dist-test-slave-nk6d krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:25:42.635130 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:42.629498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36971 (local address 127.0.98.3:59189)
0620 04:25:42.629782 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:42.629797 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:42.629915 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:42.630178 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:42.630181 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:42.630354 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:42.630455 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:42.630460 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:42.631171 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:42.631173 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:42.631760 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:42.631767 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:42.632252 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:42.632256 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:42.632341 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:42.632885 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:42.632897 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:42.634986 (+ 2089us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36971: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":72,"threads_started":1}
W20260620 04:25:42.635224 9560 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:25:42.629695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59189 (local address 127.0.98.62:36971)
0620 04:25:42.629815 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:42.629819 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:42.629858 (+ 39us) server_negotiation.cc:407] Connection header received
0620 04:25:42.629994 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:42.629996 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:42.630040 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:42.630108 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:42.630564 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:42.631073 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:42.631886 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:42.632149 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:42.632222 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:42.635120 (+ 2898us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:59189: BlockingRecv error: recv got EOF from 127.0.98.3:59189 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Jun 20 04:25:43 dist-test-slave-nk6d krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:25:43 dist-test-slave-nk6d krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:25:43.260632 9563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:43.255245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36971 (local address 127.0.98.1:42167)
0620 04:25:43.255492 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:43.255504 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:43.255590 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:43.255809 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:43.255811 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:43.256012 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:43.256117 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:43.256123 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:43.256824 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:43.256826 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:43.257408 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:43.257416 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:43.257826 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:43.257830 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:43.257902 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:43.258430 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:43.258442 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:43.260502 (+ 2060us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36971: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":70,"threads_started":1}
W20260620 04:25:43.260744 9564 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:25:43.255306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42167 (local address 127.0.98.62:36971)
0620 04:25:43.255542 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:43.255546 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:43.255560 (+ 14us) server_negotiation.cc:407] Connection header received
0620 04:25:43.255665 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:43.255668 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:43.255708 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:43.255780 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:43.256207 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:43.256740 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:43.257500 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:43.257715 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:43.257784 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:43.260639 (+ 2855us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:42167: BlockingRecv error: recv got EOF from 127.0.98.1:42167 (error 108)
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":69,"threads_started":1}
Jun 20 04:25:43 dist-test-slave-nk6d krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:25:43 dist-test-slave-nk6d krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:25:43.449730 9565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:43.444062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36971 (local address 127.0.98.2:35961)
0620 04:25:43.444372 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:43.444392 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:43.444502 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:43.444777 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:43.444780 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:43.444962 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:43.445066 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:43.445072 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:43.445946 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:43.445950 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:43.446686 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:43.446697 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:43.447229 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:43.447232 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:43.447295 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:43.447867 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:43.447879 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:43.449614 (+ 1735us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36971: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":96,"threads_started":1}
W20260620 04:25:43.449854 9564 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:25:43.444164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35961 (local address 127.0.98.62:36971)
0620 04:25:43.444339 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:43.444345 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:43.444465 (+ 120us) server_negotiation.cc:407] Connection header received
0620 04:25:43.444581 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:43.444584 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:43.444632 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:43.444714 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:43.445181 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:43.445816 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:43.446795 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:43.447125 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:43.447216 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:43.449740 (+ 2524us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:35961: BlockingRecv error: recv got EOF from 127.0.98.2:35961 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Jun 20 04:25:43 dist-test-slave-nk6d krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:25:43 dist-test-slave-nk6d krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:25:43.640771 9566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:43.635791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36971 (local address 127.0.98.3:39475)
0620 04:25:43.636078 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:43.636091 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:43.636187 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:43.636386 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:43.636387 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:43.636556 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:43.636650 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:43.636655 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:43.637338 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:43.637340 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:43.637884 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:43.637890 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:43.638298 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:43.638301 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:43.638364 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:43.638835 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:43.638849 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:43.640641 (+ 1792us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36971: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":67,"threads_started":1}
W20260620 04:25:43.640874 9564 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:25:43.635890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39475 (local address 127.0.98.62:36971)
0620 04:25:43.636036 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:43.636040 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:43.636140 (+ 100us) server_negotiation.cc:407] Connection header received
0620 04:25:43.636261 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:43.636263 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:43.636307 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:43.636364 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:43.636738 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:43.637257 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:43.637975 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:43.638200 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:43.638274 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:43.640771 (+ 2497us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:39475: BlockingRecv error: recv got EOF from 127.0.98.3:39475 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Jun 20 04:25:44 dist-test-slave-nk6d krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:25:44 dist-test-slave-nk6d krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:25:44.266563 9567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:44.261379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36971 (local address 127.0.98.1:43753)
0620 04:25:44.261625 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:44.261638 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:44.261727 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:44.261986 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:44.261989 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:44.262208 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:44.262307 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:44.262312 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:44.263072 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:44.263075 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:44.263676 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:44.263684 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:44.264100 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:44.264103 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:44.264168 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:44.264664 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:44.264676 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:44.266446 (+ 1770us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36971: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":69,"threads_started":1}
W20260620 04:25:44.266710 9568 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:25:44.261453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43753 (local address 127.0.98.62:36971)
0620 04:25:44.261716 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:44.261719 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:44.261732 (+ 13us) server_negotiation.cc:407] Connection header received
0620 04:25:44.261791 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:44.261793 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:44.261832 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:44.261905 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:44.262411 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:44.262938 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:44.263767 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:44.264003 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:44.264076 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:44.266580 (+ 2504us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:43753: BlockingRecv error: recv got EOF from 127.0.98.1:43753 (error 108)
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":88,"threads_started":1}
Jun 20 04:25:44 dist-test-slave-nk6d krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:25:44 dist-test-slave-nk6d krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:25:44.455809 9569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:44.450543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36971 (local address 127.0.98.2:35965)
0620 04:25:44.450836 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:44.450848 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:44.450947 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:44.451176 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:44.451178 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:44.451339 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:44.451432 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:44.451437 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:44.452198 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:44.452201 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:44.452800 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:44.452811 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:44.453299 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:44.453301 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:44.453366 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:44.453848 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:44.453859 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:44.455663 (+ 1804us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36971: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":87,"threads_started":1}
W20260620 04:25:44.455929 9568 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:25:44.450651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35965 (local address 127.0.98.62:36971)
0620 04:25:44.450818 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:44.450823 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:44.450918 (+ 95us) server_negotiation.cc:407] Connection header received
0620 04:25:44.451020 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:44.451022 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:44.451068 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:44.451148 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:44.451514 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:44.452089 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:44.452923 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:44.453206 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:44.453288 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:44.455808 (+ 2520us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:35965: BlockingRecv error: recv got EOF from 127.0.98.2:35965 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Jun 20 04:25:44 dist-test-slave-nk6d krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:25:44 dist-test-slave-nk6d krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:25:44.646744 9570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:44.641499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36971 (local address 127.0.98.3:35571)
0620 04:25:44.641775 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:44.641788 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:44.641892 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:44.642146 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:44.642148 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:44.642363 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:44.642483 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:44.642491 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:44.643282 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:44.643285 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:44.643837 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:44.643845 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:44.644311 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:44.644314 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:44.644380 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:44.644927 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:44.644939 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:44.646618 (+ 1679us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36971: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":85,"threads_started":1}
W20260620 04:25:44.646860 9568 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:25:44.641565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35571 (local address 127.0.98.62:36971)
0620 04:25:44.641701 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:44.641706 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:44.641872 (+ 166us) server_negotiation.cc:407] Connection header received
0620 04:25:44.641980 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:44.641983 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:44.642031 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:44.642116 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:44.642606 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:44.643158 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:44.643952 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:44.644208 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:44.644287 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:44.646751 (+ 2464us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:35571: BlockingRecv error: recv got EOF from 127.0.98.3:35571 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Jun 20 04:25:45 dist-test-slave-nk6d krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:25:45 dist-test-slave-nk6d krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:25:45.272593 9571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:45.267328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36971 (local address 127.0.98.1:48061)
0620 04:25:45.267588 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:45.267602 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:45.267686 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:45.267935 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:45.267937 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:45.268130 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:45.268230 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:45.268237 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:45.269018 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:45.269021 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:45.269670 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:45.269684 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:45.270151 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:45.270155 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:45.270240 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:45.270731 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:45.270742 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:45.272480 (+ 1738us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36971: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":90,"threads_started":1}
W20260620 04:25:45.272747 9572 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:25:45.267402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48061 (local address 127.0.98.62:36971)
0620 04:25:45.267653 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:45.267656 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:45.267674 (+ 18us) server_negotiation.cc:407] Connection header received
0620 04:25:45.267768 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:45.267770 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:45.267816 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:45.267905 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:45.268364 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:45.268912 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:45.269778 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:45.270042 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:45.270132 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:45.272613 (+ 2481us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:48061: BlockingRecv error: recv got EOF from 127.0.98.1:48061 (error 108)
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":79,"threads_started":1}
Jun 20 04:25:45 dist-test-slave-nk6d krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:25:45 dist-test-slave-nk6d krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:25:45.461412 9573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:45.456650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36971 (local address 127.0.98.2:40563)
0620 04:25:45.456904 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:45.456917 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:45.457006 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:45.457216 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:45.457219 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:45.457387 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:45.457479 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:45.457484 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:45.458152 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:45.458154 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:45.458735 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:45.458742 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:45.459137 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:45.459140 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:45.459200 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:45.459624 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:45.459636 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:45.461290 (+ 1654us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36971: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":88,"threads_started":1}
W20260620 04:25:45.461544 9572 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:25:45.456785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40563 (local address 127.0.98.62:36971)
0620 04:25:45.456904 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:45.456907 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:45.456974 (+ 67us) server_negotiation.cc:407] Connection header received
0620 04:25:45.457074 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:45.457076 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:45.457116 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:45.457181 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:45.457561 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:45.458069 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:45.458834 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:45.459049 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:45.459125 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:45.461428 (+ 2303us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:40563: BlockingRecv error: recv got EOF from 127.0.98.2:40563 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Jun 20 04:25:45 dist-test-slave-nk6d krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:25:45 dist-test-slave-nk6d krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:25:45.652699 9574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:45.647532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36971 (local address 127.0.98.3:37493)
0620 04:25:45.647782 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:45.647796 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:45.647902 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:45.648158 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:45.648161 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:45.648380 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:45.648473 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:45.648478 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:45.649237 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:45.649240 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:45.649807 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:45.649814 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:45.650235 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:45.650238 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:45.650327 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:45.650757 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:45.650768 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:45.652485 (+ 1717us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36971: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":82,"threads_started":1}
W20260620 04:25:45.652731 9572 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:25:45.647626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37493 (local address 127.0.98.62:36971)
0620 04:25:45.647761 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:45.647766 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:45.647865 (+ 99us) server_negotiation.cc:407] Connection header received
0620 04:25:45.648008 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:45.648011 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:45.648061 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:45.648126 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:45.648586 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:45.649137 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:45.649914 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:45.650139 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:45.650223 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:45.652636 (+ 2413us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:37493: BlockingRecv error: recv got EOF from 127.0.98.3:37493 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Jun 20 04:25:46 dist-test-slave-nk6d krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:25:46 dist-test-slave-nk6d krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:25:46.281952 9575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:46.273312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36971 (local address 127.0.98.1:38933)
0620 04:25:46.273557 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:46.273571 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:46.273660 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:46.273968 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:46.273971 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:46.274157 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:46.274257 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:46.274263 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:46.274992 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:46.274995 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:46.275560 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:46.275568 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:46.276092 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:46.276095 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:46.276155 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:46.279829 (+ 3674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:46.279845 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:46.281827 (+ 1982us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36971: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":75,"threads_started":1}
W20260620 04:25:46.282115 9576 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:25:46.273403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38933 (local address 127.0.98.62:36971)
0620 04:25:46.273644 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:46.273648 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:46.273662 (+ 14us) server_negotiation.cc:407] Connection header received
0620 04:25:46.273726 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:46.273728 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:46.273769 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:46.273867 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:46.274368 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:46.274908 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:46.275676 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:46.275920 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:46.276073 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:46.281990 (+ 5917us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:38933: BlockingRecv error: recv got EOF from 127.0.98.1:38933 (error 108)
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":81,"threads_started":1}
Jun 20 04:25:46 dist-test-slave-nk6d krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:25:46 dist-test-slave-nk6d krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:25:46.467374 9577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:46.462101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36971 (local address 127.0.98.2:37371)
0620 04:25:46.462354 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:46.462368 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:46.462468 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:46.462699 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:46.462701 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:46.462868 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:46.462988 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:46.462994 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:46.463738 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:46.463741 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:46.464376 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:46.464386 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:46.464787 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:46.464789 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:46.464845 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:46.465330 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:46.465341 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:46.467219 (+ 1878us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36971: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":88,"threads_started":1}
W20260620 04:25:46.467464 9576 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:25:46.462190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37371 (local address 127.0.98.62:36971)
0620 04:25:46.462372 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:46.462376 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:46.462436 (+ 60us) server_negotiation.cc:407] Connection header received
0620 04:25:46.462549 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:46.462551 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:46.462594 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:46.462683 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:46.463096 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:46.463646 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:46.464485 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:46.464700 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:46.464778 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:46.467364 (+ 2586us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:37371: BlockingRecv error: recv got EOF from 127.0.98.2:37371 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Jun 20 04:25:46 dist-test-slave-nk6d krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:25:46 dist-test-slave-nk6d krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:25:46.658707 9578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:46.653362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36971 (local address 127.0.98.3:51271)
0620 04:25:46.653655 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:46.653669 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:46.653768 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:46.654002 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:46.654005 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:46.654176 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:46.654270 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:46.654275 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:46.654989 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:46.654992 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:46.655564 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:46.655571 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:46.655999 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:46.656003 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:46.656079 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:46.656606 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:46.656623 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:46.658577 (+ 1954us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36971: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":93,"threads_started":1}
W20260620 04:25:46.658803 9576 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:25:46.653432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51271 (local address 127.0.98.62:36971)
0620 04:25:46.653549 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:46.653552 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:46.653737 (+ 185us) server_negotiation.cc:407] Connection header received
0620 04:25:46.653850 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:46.653852 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:46.653895 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:46.653963 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:46.654357 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:46.654888 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:46.655661 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:46.655874 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:46.655977 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:46.658696 (+ 2719us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:51271: BlockingRecv error: recv got EOF from 127.0.98.3:51271 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Jun 20 04:25:47 dist-test-slave-nk6d krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:25:47 dist-test-slave-nk6d krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:25:47.288245 9579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:47.282711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36971 (local address 127.0.98.1:60151)
0620 04:25:47.283059 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:47.283074 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:47.283174 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:47.283431 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:47.283434 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:47.283600 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:47.283698 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:47.283704 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:47.284467 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:47.284470 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:47.285063 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:47.285072 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:47.285526 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:47.285530 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:47.285600 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:47.286093 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:47.286107 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:47.288099 (+ 1992us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36971: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":67,"threads_started":1}
W20260620 04:25:47.288344 9580 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:25:47.282824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60151 (local address 127.0.98.62:36971)
0620 04:25:47.283070 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:47.283073 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:47.283143 (+ 70us) server_negotiation.cc:407] Connection header received
0620 04:25:47.283260 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:47.283262 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:47.283305 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:47.283385 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:47.283790 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:47.284350 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:47.285182 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:47.285406 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:47.285485 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:47.288232 (+ 2747us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:60151: BlockingRecv error: recv got EOF from 127.0.98.1:60151 (error 108)
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":92,"threads_started":1}
Jun 20 04:25:47 dist-test-slave-nk6d krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:25:47 dist-test-slave-nk6d krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:25:47.473661 9581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:47.468306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36971 (local address 127.0.98.2:42107)
0620 04:25:47.468635 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:47.468648 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:47.468770 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:47.469040 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:47.469043 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:47.469237 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:47.469334 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:47.469340 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:47.470177 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:47.470180 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:47.470786 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:47.470794 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:47.471209 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:47.471212 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:47.471273 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:47.471772 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:47.471784 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:47.473542 (+ 1758us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36971: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":92,"threads_started":1}
W20260620 04:25:47.473825 9580 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:25:47.468373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42107 (local address 127.0.98.62:36971)
0620 04:25:47.468559 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:47.468564 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:47.468726 (+ 162us) server_negotiation.cc:407] Connection header received
0620 04:25:47.468849 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:47.468851 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:47.468897 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:47.469006 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:47.469435 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:47.470066 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:47.470886 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:47.471111 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:47.471188 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:47.473707 (+ 2519us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:42107: BlockingRecv error: recv got EOF from 127.0.98.2:42107 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 20 04:25:47 dist-test-slave-nk6d krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:25:47 dist-test-slave-nk6d krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:25:47.664731 9582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:47.659441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36971 (local address 127.0.98.3:55793)
0620 04:25:47.659753 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:47.659767 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:47.659873 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:47.660158 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:47.660161 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:47.660332 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:47.660430 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:47.660435 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:47.661162 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:47.661165 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:47.661747 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:47.661755 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:47.662143 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:47.662146 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:47.662205 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:47.662726 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:47.662738 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:47.664616 (+ 1878us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36971: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":88,"threads_started":1}
W20260620 04:25:47.664896 9580 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:25:47.659533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55793 (local address 127.0.98.62:36971)
0620 04:25:47.659697 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:47.659701 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:47.659835 (+ 134us) server_negotiation.cc:407] Connection header received
0620 04:25:47.659947 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:47.659950 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:47.660020 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:47.660092 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:47.660530 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:47.661059 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:47.661837 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:47.662047 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:47.662127 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:47.664785 (+ 2658us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:55793: BlockingRecv error: recv got EOF from 127.0.98.3:55793 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Jun 20 04:25:48 dist-test-slave-nk6d krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:25:48 dist-test-slave-nk6d krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:25:48.294088 9583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:48.289023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36971 (local address 127.0.98.1:34969)
0620 04:25:48.289274 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:48.289287 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:48.289372 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:48.289644 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:48.289645 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:48.289808 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:48.289899 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:48.289904 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:48.290618 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:48.290621 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:48.291193 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:48.291201 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:48.291632 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:48.291634 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:48.291690 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:48.292172 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:48.292183 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:48.293935 (+ 1752us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36971: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":72,"threads_started":1}
W20260620 04:25:48.294204 9584 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:25:48.289222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34969 (local address 127.0.98.62:36971)
0620 04:25:48.289427 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:48.289432 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:48.289444 (+ 12us) server_negotiation.cc:407] Connection header received
0620 04:25:48.289480 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:48.289482 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:48.289523 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:48.289597 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:48.289984 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:48.290513 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:48.291294 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:48.291516 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:48.291591 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:48.294085 (+ 2494us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:34969: BlockingRecv error: recv got EOF from 127.0.98.1:34969 (error 108)
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":58,"threads_started":1}
Jun 20 04:25:48 dist-test-slave-nk6d krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:25:48 dist-test-slave-nk6d krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:25:48.479528 9585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:48.474462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36971 (local address 127.0.98.2:50535)
0620 04:25:48.474721 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:48.474734 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:48.474834 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:48.475098 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:48.475100 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:48.475273 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:48.475377 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:48.475382 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:48.476312 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:48.476315 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:48.476876 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:48.476884 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:48.477295 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:48.477297 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:48.477358 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:48.477805 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:48.477815 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:48.479407 (+ 1592us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36971: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":69,"threads_started":1}
W20260620 04:25:48.479642 9584 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:25:48.474569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50535 (local address 127.0.98.62:36971)
0620 04:25:48.474737 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:48.474741 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:48.474797 (+ 56us) server_negotiation.cc:407] Connection header received
0620 04:25:48.474903 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:48.474905 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:48.474955 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:48.475020 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:48.475468 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:48.476218 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:48.476973 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:48.477203 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:48.477277 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:48.479532 (+ 2255us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:50535: BlockingRecv error: recv got EOF from 127.0.98.2:50535 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Jun 20 04:25:48 dist-test-slave-nk6d krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:25:48 dist-test-slave-nk6d krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:25:48.670759 9586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:48.665494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36971 (local address 127.0.98.3:37387)
0620 04:25:48.665763 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:48.665775 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:48.665871 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:48.666101 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:48.666104 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:48.666281 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:48.666399 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:48.666405 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:48.667163 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:48.667165 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:48.667740 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:48.667748 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:48.668152 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:48.668154 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:48.668214 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:48.668675 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:48.668687 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:48.670567 (+ 1880us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36971: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":83,"threads_started":1}
W20260620 04:25:48.670887 9584 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:25:48.665567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37387 (local address 127.0.98.62:36971)
0620 04:25:48.665723 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:48.665727 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:48.665840 (+ 113us) server_negotiation.cc:407] Connection header received
0620 04:25:48.665955 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:48.665958 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:48.666008 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:48.666079 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:48.666481 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:48.667074 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:48.667825 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:48.668073 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:48.668146 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:48.670770 (+ 2624us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:37387: BlockingRecv error: recv got EOF from 127.0.98.3:37387 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Jun 20 04:25:49 dist-test-slave-nk6d krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:25:49 dist-test-slave-nk6d krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:25:49.300386 9587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:49.294997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36971 (local address 127.0.98.1:50631)
0620 04:25:49.295264 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:49.295278 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:49.295381 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:49.295622 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:49.295624 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:49.295833 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:49.295939 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:49.295945 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:49.296830 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:49.296833 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:49.297435 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:49.297444 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:49.297907 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:49.297910 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:49.297973 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:49.298450 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:49.298461 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:49.300258 (+ 1797us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36971: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":82,"threads_started":1}
W20260620 04:25:49.300496 9588 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:25:49.295054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50631 (local address 127.0.98.62:36971)
0620 04:25:49.295285 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:49.295288 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:49.295343 (+ 55us) server_negotiation.cc:407] Connection header received
0620 04:25:49.295454 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:49.295456 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:49.295501 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:49.295574 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:49.296112 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:49.296679 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:49.297564 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:49.297810 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:49.297889 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:49.300383 (+ 2494us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:50631: BlockingRecv error: recv got EOF from 127.0.98.1:50631 (error 108)
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":72,"threads_started":1}
Jun 20 04:25:49 dist-test-slave-nk6d krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:25:49 dist-test-slave-nk6d krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:25:49.485486 9589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:49.480211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36971 (local address 127.0.98.2:49837)
0620 04:25:49.480494 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:49.480508 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:49.480611 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:49.480836 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:49.480838 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:49.481044 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:49.481157 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:49.481163 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:49.481882 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:49.481884 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:49.482453 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:49.482460 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:49.482902 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:49.482905 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:49.482964 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:49.483419 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:49.483430 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:49.485352 (+ 1922us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36971: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":97,"threads_started":1}
W20260620 04:25:49.485600 9588 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:25:49.480340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49837 (local address 127.0.98.62:36971)
0620 04:25:49.480498 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:49.480501 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:49.480574 (+ 73us) server_negotiation.cc:407] Connection header received
0620 04:25:49.480680 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:49.480682 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:49.480726 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:49.480817 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:49.481254 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:49.481781 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:49.482558 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:49.482802 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:49.482879 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:49.485497 (+ 2618us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:49837: BlockingRecv error: recv got EOF from 127.0.98.2:49837 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Jun 20 04:25:49 dist-test-slave-nk6d krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:25:49 dist-test-slave-nk6d krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:25:49.676613 9590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:49.671428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36971 (local address 127.0.98.3:34265)
0620 04:25:49.671709 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:49.671723 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:49.671832 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:49.672094 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:49.672096 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:49.672293 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:49.672384 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:49.672390 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:49.673143 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:49.673146 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:49.673694 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:49.673701 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:49.674129 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:49.674133 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:49.674210 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:49.674679 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:49.674689 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:49.676498 (+ 1809us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36971: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":97,"threads_started":1}
W20260620 04:25:49.676774 9588 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:25:49.671503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34265 (local address 127.0.98.62:36971)
0620 04:25:49.671681 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:49.671685 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:49.671793 (+ 108us) server_negotiation.cc:407] Connection header received
0620 04:25:49.671919 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:49.671922 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:49.671989 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:49.672053 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:49.672497 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:49.673024 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:49.673806 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:49.674021 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:49.674106 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:49.676646 (+ 2540us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:34265: BlockingRecv error: recv got EOF from 127.0.98.3:34265 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 20 04:25:50 dist-test-slave-nk6d krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:25:50 dist-test-slave-nk6d krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:25:50.306659 9591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:50.301315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36971 (local address 127.0.98.1:38363)
0620 04:25:50.301681 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:50.301696 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:50.301818 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:50.302105 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:50.302108 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:50.302321 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:50.302439 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:50.302447 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:50.303166 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:50.303169 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:50.303723 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:50.303730 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:50.304137 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:50.304140 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:50.304206 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:50.304707 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:50.304719 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:50.306512 (+ 1793us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36971: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":79,"threads_started":1}
W20260620 04:25:50.306782 9592 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:25:50.301422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38363 (local address 127.0.98.62:36971)
0620 04:25:50.301684 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:50.301688 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:50.301780 (+ 92us) server_negotiation.cc:407] Connection header received
0620 04:25:50.301917 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:50.301920 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:50.301974 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:50.302058 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:50.302553 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:50.303076 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:50.303815 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:50.304051 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:50.304120 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:50.306666 (+ 2546us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:38363: BlockingRecv error: recv got EOF from 127.0.98.1:38363 (error 108)
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":69,"threads_started":1}
Jun 20 04:25:50 dist-test-slave-nk6d krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:25:50 dist-test-slave-nk6d krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:25:50.491807 9593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:50.486339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36971 (local address 127.0.98.2:37461)
0620 04:25:50.486611 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:50.486625 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:50.486717 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:50.486935 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:50.486937 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:50.487108 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:50.487202 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:50.487207 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:50.488017 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:50.488020 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:50.488647 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:50.488657 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:50.489223 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:50.489226 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:50.489300 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:50.489804 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:50.489814 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:50.491643 (+ 1829us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36971: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":87,"threads_started":1}
W20260620 04:25:50.491919 9592 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:25:50.486437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37461 (local address 127.0.98.62:36971)
0620 04:25:50.486570 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:50.486574 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:50.486679 (+ 105us) server_negotiation.cc:407] Connection header received
0620 04:25:50.486785 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:50.486787 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:50.486826 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:50.486908 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:50.487297 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:50.487867 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:50.488760 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:50.489099 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:50.489219 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:50.491810 (+ 2591us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:37461: BlockingRecv error: recv got EOF from 127.0.98.2:37461 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Jun 20 04:25:50 dist-test-slave-nk6d krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:25:50 dist-test-slave-nk6d krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:25:50.682421 9594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:50.677313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36971 (local address 127.0.98.3:52837)
0620 04:25:50.677571 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:50.677583 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:50.677674 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:50.677868 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:50.677876 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:50.678044 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:50.678136 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:50.678141 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:50.678871 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:50.678875 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:50.679538 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:50.679548 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:50.680035 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:50.680039 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:50.680104 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:50.680599 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:50.680611 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:50.682307 (+ 1696us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36971: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":84,"threads_started":1}
W20260620 04:25:50.682540 9592 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:25:50.677448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52837 (local address 127.0.98.62:36971)
0620 04:25:50.677568 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:50.677572 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:50.677639 (+ 67us) server_negotiation.cc:407] Connection header received
0620 04:25:50.677739 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:50.677742 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:50.677784 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:50.677845 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:50.678221 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:50.678740 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:50.679649 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:50.679897 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:50.680014 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:50.682437 (+ 2423us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:52837: BlockingRecv error: recv got EOF from 127.0.98.3:52837 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jun 20 04:25:51 dist-test-slave-nk6d krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:25:51 dist-test-slave-nk6d krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:25:51.316224 9595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:51.307436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36971 (local address 127.0.98.1:35923)
0620 04:25:51.307721 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:51.307735 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:51.307817 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:51.308107 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:51.308109 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:51.308276 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:51.308371 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:51.308376 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:51.309141 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:51.309143 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:51.309704 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:51.309711 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:51.310156 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:51.310159 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:51.310217 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:51.314209 (+ 3992us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:51.314221 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:51.316110 (+ 1889us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36971: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":84,"threads_started":1}
W20260620 04:25:51.316423 9596 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:25:51.307505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35923 (local address 127.0.98.62:36971)
0620 04:25:51.307758 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:51.307761 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:51.307791 (+ 30us) server_negotiation.cc:407] Connection header received
0620 04:25:51.307932 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:51.307934 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:51.307999 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:51.308065 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:51.308501 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:51.309054 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:51.309805 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:51.310068 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:51.310149 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:51.316264 (+ 6115us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:35923: BlockingRecv error: recv got EOF from 127.0.98.1:35923 (error 108)
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":81,"threads_started":1}
Jun 20 04:25:51 dist-test-slave-nk6d krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:25:51 dist-test-slave-nk6d krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:25:51.498112 9597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:51.492670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36971 (local address 127.0.98.2:60797)
0620 04:25:51.492950 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:51.493000 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:51.493112 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:51.493357 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:51.493359 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:51.493545 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:51.493634 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:51.493639 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:51.494371 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:51.494375 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:51.494948 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:51.494957 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:51.495400 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:51.495403 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:51.495483 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:51.496006 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:51.496021 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:51.497995 (+ 1974us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36971: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":82,"threads_started":1}
W20260620 04:25:51.498252 9596 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:25:51.492785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60797 (local address 127.0.98.62:36971)
0620 04:25:51.492952 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:51.492956 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:51.493074 (+ 118us) server_negotiation.cc:407] Connection header received
0620 04:25:51.493199 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:51.493202 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:51.493251 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:51.493315 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:51.493752 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:51.494269 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:51.495073 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:51.495299 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:51.495382 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:51.498146 (+ 2764us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:60797: BlockingRecv error: recv got EOF from 127.0.98.2:60797 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Jun 20 04:25:51 dist-test-slave-nk6d krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:25:51 dist-test-slave-nk6d krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:25:51.688506 9598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:51.683123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36971 (local address 127.0.98.3:56299)
0620 04:25:51.683409 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:51.683421 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:51.683554 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:51.683795 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:51.683798 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:51.684024 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:51.684123 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:51.684128 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:51.684848 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:51.684852 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:51.685407 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:51.685414 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:51.685833 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:51.685836 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:51.685913 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:51.686409 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:51.686420 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:51.688340 (+ 1920us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36971: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":74,"threads_started":1}
W20260620 04:25:51.688601 9596 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:25:51.683145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56299 (local address 127.0.98.62:36971)
0620 04:25:51.683265 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:51.683269 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:51.683494 (+ 225us) server_negotiation.cc:407] Connection header received
0620 04:25:51.683644 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:51.683648 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:51.683695 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:51.683757 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:51.684235 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:51.684746 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:51.685518 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:51.685734 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:51.685818 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:51.688491 (+ 2673us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:56299: BlockingRecv error: recv got EOF from 127.0.98.3:56299 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Jun 20 04:25:52 dist-test-slave-nk6d krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:25:52 dist-test-slave-nk6d krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:25:52.322314 9599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:52.316935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36971 (local address 127.0.98.1:60357)
0620 04:25:52.317190 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:52.317205 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:52.317304 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:52.317516 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:52.317518 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:52.317702 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:52.317799 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:52.317805 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:52.318491 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:52.318494 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:52.319095 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:52.319107 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:52.319650 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:52.319654 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:52.319736 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:52.320265 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:52.320281 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:52.322197 (+ 1916us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36971: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":78,"threads_started":1}
W20260620 04:25:52.322466 9600 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:25:52.317003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60357 (local address 127.0.98.62:36971)
0620 04:25:52.317224 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:52.317228 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:52.317256 (+ 28us) server_negotiation.cc:407] Connection header received
0620 04:25:52.317374 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:52.317376 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:52.317418 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:52.317492 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:52.317885 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:52.318398 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:52.319223 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:52.319531 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:52.319646 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:52.322335 (+ 2689us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:60357: BlockingRecv error: recv got EOF from 127.0.98.1:60357 (error 108)
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":67,"threads_started":1}
Jun 20 04:25:52 dist-test-slave-nk6d krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:25:52 dist-test-slave-nk6d krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:25:52.504424 9601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:52.498853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36971 (local address 127.0.98.2:41723)
0620 04:25:52.499142 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:52.499157 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:52.499276 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:52.499530 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:52.499533 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:52.499831 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:52.499934 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:52.499941 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:52.500809 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:52.500812 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:52.501430 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:52.501438 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:52.501954 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:52.501957 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:52.502059 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:52.502532 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:52.502545 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:52.504307 (+ 1762us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36971: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":84,"threads_started":1}
W20260620 04:25:52.504544 9600 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:25:52.498953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41723 (local address 127.0.98.62:36971)
0620 04:25:52.499125 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:52.499130 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:52.499232 (+ 102us) server_negotiation.cc:407] Connection header received
0620 04:25:52.499367 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:52.499371 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:52.499417 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:52.499503 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:52.500124 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:52.500689 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:52.501569 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:52.501799 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:52.501873 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:52.504439 (+ 2566us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:41723: BlockingRecv error: recv got EOF from 127.0.98.2:41723 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Jun 20 04:25:52 dist-test-slave-nk6d krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:25:52 dist-test-slave-nk6d krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:25:52.694957 9602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:52.689287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36971 (local address 127.0.98.3:58257)
0620 04:25:52.689574 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:52.689588 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:52.689698 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:52.689924 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:52.689926 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:52.690106 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:52.690201 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:52.690207 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:52.690968 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:52.690970 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:52.691560 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:52.691569 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:52.692181 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:52.692184 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:52.692272 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:52.692800 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:52.692813 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:52.694808 (+ 1995us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36971: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":82,"threads_started":1}
W20260620 04:25:52.695010 9600 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:25:52.689399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58257 (local address 127.0.98.62:36971)
0620 04:25:52.689556 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:52.689559 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:52.689653 (+ 94us) server_negotiation.cc:407] Connection header received
0620 04:25:52.689782 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:52.689784 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:52.689831 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:52.689897 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:52.690334 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:52.690858 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:52.691694 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:52.691926 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:52.692110 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:52.694916 (+ 2806us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:58257: BlockingRecv error: recv got EOF from 127.0.98.3:58257 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Jun 20 04:25:53 dist-test-slave-nk6d krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:25:53 dist-test-slave-nk6d krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:25:53.328506 9603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:53.323088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36971 (local address 127.0.98.1:56795)
0620 04:25:53.323364 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:53.323381 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:53.323483 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:53.323691 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:53.323692 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:53.323882 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:53.324017 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:53.324025 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:53.324790 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:53.324793 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:53.325352 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:53.325359 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:53.325777 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:53.325780 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:53.325842 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:53.326369 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:53.326381 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:53.328383 (+ 2002us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36971: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":100,"threads_started":1}
W20260620 04:25:53.328635 9604 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:25:53.323214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56795 (local address 127.0.98.62:36971)
0620 04:25:53.323485 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:53.323488 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:53.323503 (+ 15us) server_negotiation.cc:407] Connection header received
0620 04:25:53.323544 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:53.323546 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:53.323587 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:53.323664 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:53.324131 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:53.324663 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:53.325442 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:53.325657 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:53.325746 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:53.328506 (+ 2760us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:56795: BlockingRecv error: recv got EOF from 127.0.98.1:56795 (error 108)
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":74,"threads_started":1}
Jun 20 04:25:53 dist-test-slave-nk6d krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:25:53 dist-test-slave-nk6d krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:25:53.510166 9605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:53.505153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36971 (local address 127.0.98.2:59551)
0620 04:25:53.505410 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:53.505424 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:53.505517 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:53.505753 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:53.505755 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:53.505926 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:53.506022 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:53.506028 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:53.506745 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:53.506748 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:53.507317 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:53.507326 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:53.507755 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:53.507757 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:53.507815 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:53.508354 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:53.508367 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:53.510047 (+ 1680us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36971: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":108,"threads_started":1}
W20260620 04:25:53.510390 9604 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:25:53.505295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59551 (local address 127.0.98.62:36971)
0620 04:25:53.505427 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:53.505430 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:53.505482 (+ 52us) server_negotiation.cc:407] Connection header received
0620 04:25:53.505586 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:53.505588 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:53.505631 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:53.505698 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:53.506129 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:53.506646 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:53.507441 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:53.507666 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:53.507751 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:53.510185 (+ 2434us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:59551: BlockingRecv error: recv got EOF from 127.0.98.2:59551 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Jun 20 04:25:53 dist-test-slave-nk6d krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:25:53 dist-test-slave-nk6d krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:25:53.701786 9606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:53.695697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36971 (local address 127.0.98.3:51453)
0620 04:25:53.695942 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:53.695976 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:53.696085 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:53.696354 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:53.696357 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:53.696521 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:53.696611 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:53.696616 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:53.697346 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:53.697348 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:53.697896 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:53.697903 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:53.698303 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:53.698306 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:53.698367 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:53.698869 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:53.698880 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:53.701667 (+ 2787us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36971: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":75,"threads_started":1}
W20260620 04:25:53.701901 9604 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:25:53.695724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51453 (local address 127.0.98.62:36971)
0620 04:25:53.695855 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:53.695860 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:53.696052 (+ 192us) server_negotiation.cc:407] Connection header received
0620 04:25:53.696201 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:53.696204 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:53.696253 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:53.696326 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:53.696732 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:53.697258 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:53.697989 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:53.698206 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:53.698293 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:53.701799 (+ 3506us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:51453: BlockingRecv error: recv got EOF from 127.0.98.3:51453 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Jun 20 04:25:54 dist-test-slave-nk6d krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:25:54 dist-test-slave-nk6d krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:25:54.334606 9607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:54.329264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36971 (local address 127.0.98.1:44145)
0620 04:25:54.329570 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:54.329588 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:54.329701 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:54.329899 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:54.329901 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:54.330112 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:54.330273 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:54.330280 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:54.331137 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:54.331140 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:54.331751 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:54.331760 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:54.332300 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:54.332304 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:54.332377 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:54.332823 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:54.332834 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:54.334494 (+ 1660us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36971: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":79,"threads_started":1}
W20260620 04:25:54.334724 9608 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:25:54.329393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44145 (local address 127.0.98.62:36971)
0620 04:25:54.329704 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:54.329707 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:54.329721 (+ 14us) server_negotiation.cc:407] Connection header received
0620 04:25:54.329777 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:54.329779 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:54.329819 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:54.329891 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:54.330397 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:54.331022 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:54.331853 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:54.332164 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:54.332252 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:54.334615 (+ 2363us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:44145: BlockingRecv error: recv got EOF from 127.0.98.1:44145 (error 108)
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":92,"threads_started":1}
Jun 20 04:25:54 dist-test-slave-nk6d krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:25:54 dist-test-slave-nk6d krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:25:54.516616 9609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:54.510988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36971 (local address 127.0.98.2:43023)
0620 04:25:54.511266 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:54.511279 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:54.511383 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:54.511646 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:54.511648 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:54.511827 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:54.511941 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:54.511947 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:54.512781 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:54.512783 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:54.513356 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:54.513364 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:54.513801 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:54.513803 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:54.513882 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:54.514384 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:54.514394 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:54.516498 (+ 2104us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36971: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":90,"threads_started":1}
W20260620 04:25:54.516736 9608 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:25:54.511087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43023 (local address 127.0.98.62:36971)
0620 04:25:54.511249 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:54.511253 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:54.511348 (+ 95us) server_negotiation.cc:407] Connection header received
0620 04:25:54.511459 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:54.511462 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:54.511505 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:54.511574 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:54.512090 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:54.512676 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:54.513479 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:54.513700 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:54.513786 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:54.516624 (+ 2838us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:43023: BlockingRecv error: recv got EOF from 127.0.98.2:43023 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 20 04:25:54 dist-test-slave-nk6d krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:25:54 dist-test-slave-nk6d krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:25:54.707638 9610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:54.702552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36971 (local address 127.0.98.3:55653)
0620 04:25:54.702830 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:54.702849 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:54.702948 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:54.703205 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:54.703208 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:54.703378 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:54.703470 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:54.703476 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:54.704316 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:54.704318 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:54.704877 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:54.704886 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:54.705370 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:54.705372 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:54.705429 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:54.705895 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:54.705906 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:54.707504 (+ 1598us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36971: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":73,"threads_started":1}
W20260620 04:25:54.707782 9608 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:25:54.702629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55653 (local address 127.0.98.62:36971)
0620 04:25:54.702782 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:54.702786 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:54.702946 (+ 160us) server_negotiation.cc:407] Connection header received
0620 04:25:54.703022 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:54.703025 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:54.703071 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:54.703140 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:54.703579 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:54.704217 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:54.705016 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:54.705236 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:54.705359 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:54.707661 (+ 2302us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:55653: BlockingRecv error: recv got EOF from 127.0.98.3:55653 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 20 04:25:55 dist-test-slave-nk6d krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:25:55 dist-test-slave-nk6d krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:25:55.340955 9611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:55.335431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36971 (local address 127.0.98.1:52915)
0620 04:25:55.335722 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:55.335738 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:55.335821 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:55.336124 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:55.336127 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:55.336302 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:55.336404 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:55.336409 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:55.337145 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:55.337148 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:55.337729 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:55.337737 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:55.338214 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:55.338217 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:55.338286 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:55.338786 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:55.338798 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:55.340791 (+ 1993us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36971: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":76,"threads_started":1}
W20260620 04:25:55.341094 9612 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:25:55.335521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52915 (local address 127.0.98.62:36971)
0620 04:25:55.335768 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:55.335771 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:55.335783 (+ 12us) server_negotiation.cc:407] Connection header received
0620 04:25:55.335911 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:55.335913 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:55.335982 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:55.336056 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:55.336505 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:55.337029 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:55.337846 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:55.338087 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:55.338164 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:55.340941 (+ 2777us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:52915: BlockingRecv error: recv got EOF from 127.0.98.1:52915 (error 108)
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":77,"threads_started":1}
Jun 20 04:25:55 dist-test-slave-nk6d krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:25:55 dist-test-slave-nk6d krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:25:55.522891 9613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:55.517484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36971 (local address 127.0.98.2:51361)
0620 04:25:55.517757 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:55.517771 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:55.517869 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:55.518107 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:55.518109 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:55.518278 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:55.518422 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:55.518428 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:55.519353 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:55.519356 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:55.520239 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:55.520249 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:55.520650 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:55.520652 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:55.520710 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:55.521157 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:55.521168 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:55.522766 (+ 1598us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36971: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":75,"threads_started":1}
W20260620 04:25:55.522977 9612 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:25:55.517614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51361 (local address 127.0.98.62:36971)
0620 04:25:55.517754 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:55.517758 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:55.517835 (+ 77us) server_negotiation.cc:407] Connection header received
0620 04:25:55.517945 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:55.517947 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:55.517989 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:55.518077 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:55.518533 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:55.519233 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:55.520344 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:55.520562 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:55.520639 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:55.522884 (+ 2245us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:51361: BlockingRecv error: recv got EOF from 127.0.98.2:51361 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 04:25:55.611419 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 9136
I20260620 04:25:55.616561 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 9288
I20260620 04:25:55.621307 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 9421
I20260620 04:25:55.625505 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 9066
2026-06-20T04:25:55Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17798 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-20T04:25:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:25:55Z Disabled control of system clock
I20260620 04:25:55.652650 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929496330398-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929496330398-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929496330398-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41201
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:39507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41201
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 04:25:55.759063 9617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:25:55.759330 9617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:25:55.759363 9617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:25:55.763348 9617 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:25:55.763428 9617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:25:55.763443 9617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:25:55.763460 9617 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:25:55.763474 9617 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:25:55.768350 9617 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929496330398-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929496330398-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41201
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41201
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9617
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929496330398-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:25:55.769447 9617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:25:55.770853 9617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 04:25:55.779398 9617 server_base.cc:1034] running on GCE node
W20260620 04:25:55.779359 9625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:25:55.779361 9622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:25:55.779353 9623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:25:55.780254 9617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:25:55.781328 9617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:25:55.782490 9617 hybrid_clock.cc:648] HybridClock initialized: now 1781929555782477 us; error 28 us; skew 500 ppm
I20260620 04:25:55.784297 9617 webserver.cc:460] Webserver started at http://127.0.98.62:39463/ using document root <none> and password file <none>
I20260620 04:25:55.784830 9617 fs_manager.cc:359] Metadata directory not provided
I20260620 04:25:55.784896 9617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:25:55.785104 9617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:25:55.787065 9617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929496330398-392-0/minicluster-data/master-0/data/instance:
uuid: "5457aece34ff45aabc062586ba7e04da"
format_stamp: "Formatted at 2026-06-20 04:25:55 on dist-test-slave-nk6d"
server_key: "3f7bbe5d0d0beb23d0ef9d3d23713372"
server_key_iv: "f4877c08cbcf6190b46e18c828a4d556"
server_key_version: "encryptionkey@0"
I20260620 04:25:55.787621 9617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929496330398-392-0/minicluster-data/master-0/wal/instance:
uuid: "5457aece34ff45aabc062586ba7e04da"
format_stamp: "Formatted at 2026-06-20 04:25:55 on dist-test-slave-nk6d"
server_key: "3f7bbe5d0d0beb23d0ef9d3d23713372"
server_key_iv: "f4877c08cbcf6190b46e18c828a4d556"
server_key_version: "encryptionkey@0"
I20260620 04:25:55.791134 9617 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 04:25:55.793437 9631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:55.794591 9617 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 04:25:55.794694 9617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929496330398-392-0/minicluster-data/master-0/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929496330398-392-0/minicluster-data/master-0/wal
uuid: "5457aece34ff45aabc062586ba7e04da"
format_stamp: "Formatted at 2026-06-20 04:25:55 on dist-test-slave-nk6d"
server_key: "3f7bbe5d0d0beb23d0ef9d3d23713372"
server_key_iv: "f4877c08cbcf6190b46e18c828a4d556"
server_key_version: "encryptionkey@0"
I20260620 04:25:55.794782 9617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929496330398-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929496330398-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929496330398-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:25:55.821611 9617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:25:55.822314 9617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:25:55.822492 9617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:25:55.828948 9617 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41201
I20260620 04:25:55.828961 9683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41201 every 8 connection(s)
I20260620 04:25:55.829843 9617 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:25:55.832796 9684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:25:55.837988 9684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5457aece34ff45aabc062586ba7e04da: Bootstrap starting.
I20260620 04:25:55.838032 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 9617
I20260620 04:25:55.838104 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929496330398-392-0/minicluster-data/master-0/wal/instance
I20260620 04:25:55.838351 392 external_mini_cluster.cc:1428] Setting key 155194772721c109fac5b717095b1958
I20260620 04:25:55.840513 9684 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5457aece34ff45aabc062586ba7e04da: Neither blocks nor log segments found. Creating new log.
I20260620 04:25:55.841216 9684 log.cc:824] T 00000000000000000000000000000000 P 5457aece34ff45aabc062586ba7e04da: Log is configured to *not* fsync() on all Append() calls
I20260620 04:25:55.843315 9684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5457aece34ff45aabc062586ba7e04da: No bootstrap required, opened a new log
I20260620 04:25:55.847592 9684 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5457aece34ff45aabc062586ba7e04da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5457aece34ff45aabc062586ba7e04da" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41201 } }
I20260620 04:25:55.847847 9684 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5457aece34ff45aabc062586ba7e04da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:25:55.847908 9684 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5457aece34ff45aabc062586ba7e04da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5457aece34ff45aabc062586ba7e04da, State: Initialized, Role: FOLLOWER
I20260620 04:25:55.848181 9687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:55.839661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49144 (local address 127.0.98.62:41201)
0620 04:25:55.840314 (+ 653us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:55.840328 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:55.840365 (+ 37us) server_negotiation.cc:407] Connection header received
0620 04:25:55.841028 (+ 663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:55.841047 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:55.841481 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:55.841764 (+ 283us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:55.842177 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:55.843553 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:55.844455 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:55.845152 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:55.845311 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:55.846075 (+ 764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:55.846081 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:55.846098 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:25:55.846620 (+ 522us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:55.846692 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:25:55.846721 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:55.846814 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:55.846916 (+ 102us) server_negotiation.cc:299] Negotiation successful
0620 04:25:55.847189 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":122,"threads_started":1}
I20260620 04:25:55.848387 9684 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5457aece34ff45aabc062586ba7e04da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5457aece34ff45aabc062586ba7e04da" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41201 } }
I20260620 04:25:55.848546 9684 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5457aece34ff45aabc062586ba7e04da [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:25:55.848609 9684 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5457aece34ff45aabc062586ba7e04da [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:25:55.848699 9684 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5457aece34ff45aabc062586ba7e04da [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:25:55.849550 9684 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5457aece34ff45aabc062586ba7e04da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5457aece34ff45aabc062586ba7e04da" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41201 } }
I20260620 04:25:55.849872 9684 leader_election.cc:302] T 00000000000000000000000000000000 P 5457aece34ff45aabc062586ba7e04da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5457aece34ff45aabc062586ba7e04da; no voters:
I20260620 04:25:55.850199 9684 leader_election.cc:288] T 00000000000000000000000000000000 P 5457aece34ff45aabc062586ba7e04da [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:25:55.850281 9689 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5457aece34ff45aabc062586ba7e04da [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:25:55.850533 9689 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5457aece34ff45aabc062586ba7e04da [term 1 LEADER]: Becoming Leader. State: Replica: 5457aece34ff45aabc062586ba7e04da, State: Running, Role: LEADER
I20260620 04:25:55.850836 9689 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5457aece34ff45aabc062586ba7e04da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5457aece34ff45aabc062586ba7e04da" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41201 } }
I20260620 04:25:55.851328 9684 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5457aece34ff45aabc062586ba7e04da [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:25:55.852288 9690 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5457aece34ff45aabc062586ba7e04da [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5457aece34ff45aabc062586ba7e04da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5457aece34ff45aabc062586ba7e04da" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41201 } } }
I20260620 04:25:55.852402 9690 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5457aece34ff45aabc062586ba7e04da [sys.catalog]: This master's current role is: LEADER
I20260620 04:25:55.852638 9691 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5457aece34ff45aabc062586ba7e04da [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5457aece34ff45aabc062586ba7e04da. Latest consensus state: current_term: 1 leader_uuid: "5457aece34ff45aabc062586ba7e04da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5457aece34ff45aabc062586ba7e04da" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41201 } } }
I20260620 04:25:55.852720 9691 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5457aece34ff45aabc062586ba7e04da [sys.catalog]: This master's current role is: LEADER
I20260620 04:25:55.853097 9698 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:25:55.855686 9698 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:25:55.860396 9698 catalog_manager.cc:1347] Generated new cluster ID: e89e8cf5c94d4f7792b1b7ddb6f84164
I20260620 04:25:55.860467 9698 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:25:55.873930 9698 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:25:55.874899 9698 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:25:55.879721 9698 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5457aece34ff45aabc062586ba7e04da: Generated new TSK 0
I20260620 04:25:55.880414 9698 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 04:25:55.890440 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929496330398-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929496330398-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929496330398-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929496330398-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41201
--builtin_ntp_servers=127.0.98.20:39507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260620 04:25:56.000093 9708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:25:56.000352 9708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:25:56.000388 9708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:25:56.004428 9708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:25:56.004490 9708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:25:56.004544 9708 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 04:25:56.009230 9708 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929496330398-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929496330398-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929496330398-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41201
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9708
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929496330398-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:25:56.010303 9708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:25:56.011653 9708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:25:56.020843 9713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:25:56.020798 9716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:25:56.020814 9714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:25:56.021034 9708 server_base.cc:1034] running on GCE node
I20260620 04:25:56.021502 9708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:25:56.022145 9708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:25:56.023327 9708 hybrid_clock.cc:648] HybridClock initialized: now 1781929556023309 us; error 33 us; skew 500 ppm
I20260620 04:25:56.025184 9708 webserver.cc:460] Webserver started at http://127.0.98.1:39439/ using document root <none> and password file <none>
I20260620 04:25:56.025775 9708 fs_manager.cc:359] Metadata directory not provided
I20260620 04:25:56.025841 9708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:25:56.026024 9708 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:25:56.028074 9708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929496330398-392-0/minicluster-data/ts-0/data/instance:
uuid: "b3746a3271a242779ef47150add53581"
format_stamp: "Formatted at 2026-06-20 04:25:56 on dist-test-slave-nk6d"
server_key: "5e346ef360ff11952b44cd8cb3215f87"
server_key_iv: "7ba89e7c4f9e8384286a4fc62a6960c0"
server_key_version: "encryptionkey@0"
I20260620 04:25:56.028601 9708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929496330398-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b3746a3271a242779ef47150add53581"
format_stamp: "Formatted at 2026-06-20 04:25:56 on dist-test-slave-nk6d"
server_key: "5e346ef360ff11952b44cd8cb3215f87"
server_key_iv: "7ba89e7c4f9e8384286a4fc62a6960c0"
server_key_version: "encryptionkey@0"
I20260620 04:25:56.032152 9708 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 04:25:56.034652 9722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:56.035909 9708 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:25:56.036041 9708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929496330398-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929496330398-392-0/minicluster-data/ts-0/wal
uuid: "b3746a3271a242779ef47150add53581"
format_stamp: "Formatted at 2026-06-20 04:25:56 on dist-test-slave-nk6d"
server_key: "5e346ef360ff11952b44cd8cb3215f87"
server_key_iv: "7ba89e7c4f9e8384286a4fc62a6960c0"
server_key_version: "encryptionkey@0"
I20260620 04:25:56.036139 9708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929496330398-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929496330398-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929496330398-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:25:56.054426 9708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:25:56.055112 9708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:25:56.055282 9708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:25:56.055871 9708 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:25:56.056862 9708 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:25:56.056907 9708 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:56.056959 9708 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:25:56.056988 9708 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:56.065315 9708 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42201
I20260620 04:25:56.065346 9835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42201 every 8 connection(s)
I20260620 04:25:56.066250 9708 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929496330398-392-0/minicluster-data/ts-0/data/info.pb
I20260620 04:25:56.075089 9687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:56.068038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50633 (local address 127.0.98.62:41201)
0620 04:25:56.068173 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:56.068177 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:56.069216 (+ 1039us) server_negotiation.cc:407] Connection header received
0620 04:25:56.070597 (+ 1381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:56.070600 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:56.070656 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:56.070726 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:56.071609 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:56.072177 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:56.073276 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:56.073496 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:56.073583 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:56.074636 (+ 1053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:56.074639 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:56.074641 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:25:56.074652 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:56.074669 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:25:56.074677 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:56.074733 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:56.074883 (+ 150us) server_negotiation.cc:299] Negotiation successful
0620 04:25:56.074939 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 04:25:56.075903 9838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:56.068422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41201 (local address 127.0.98.1:50633)
0620 04:25:56.069069 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:56.069106 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:56.070380 (+ 1274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:56.070900 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:56.070907 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:56.070992 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:56.071459 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:56.071471 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:56.072291 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:56.072293 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:56.073153 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:56.073162 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:56.073634 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:56.073638 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:56.073814 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:56.074378 (+ 564us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 04:25:56.074379 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:56.074419 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 04:25:56.074480 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:25:56.074746 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:25:56.074755 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:25:56.074760 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 04:25:56.074868 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 04:25:56.075131 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":115,"threads_started":1}
I20260620 04:25:56.076375 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 9708
I20260620 04:25:56.076471 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929496330398-392-0/minicluster-data/ts-0/wal/instance
I20260620 04:25:56.076735 392 external_mini_cluster.cc:1428] Setting key 741e44d94ad53bbf016ee7a6990b75ad
I20260620 04:25:56.077220 9836 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41201
I20260620 04:25:56.077530 9836 heartbeater.cc:464] Registering TS with master...
I20260620 04:25:56.078097 9836 heartbeater.cc:511] Master 127.0.98.62:41201 requested a full tablet report, sending...
I20260620 04:25:56.078555 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929496330398-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929496330398-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929496330398-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929496330398-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41201
--builtin_ntp_servers=127.0.98.20:39507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 04:25:56.079685 9648 ts_manager.cc:194] Registered new tserver with Master: b3746a3271a242779ef47150add53581 (127.0.98.1:42201)
I20260620 04:25:56.080938 9648 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:50633
W20260620 04:25:56.186390 9839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:25:56.186651 9839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:25:56.186686 9839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:25:56.190678 9839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:25:56.190738 9839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:25:56.190791 9839 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 04:25:56.195425 9839 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929496330398-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929496330398-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929496330398-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41201
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9839
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929496330398-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:25:56.196455 9839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:25:56.197736 9839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:25:56.206933 9844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:25:56.207024 9845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:25:56.206933 9847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:25:56.207363 9839 server_base.cc:1034] running on GCE node
I20260620 04:25:56.207736 9839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:25:56.208391 9839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:25:56.209555 9839 hybrid_clock.cc:648] HybridClock initialized: now 1781929556209506 us; error 59 us; skew 500 ppm
I20260620 04:25:56.211432 9839 webserver.cc:460] Webserver started at http://127.0.98.2:43521/ using document root <none> and password file <none>
I20260620 04:25:56.212039 9839 fs_manager.cc:359] Metadata directory not provided
I20260620 04:25:56.212121 9839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:25:56.212324 9839 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:25:56.214335 9839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929496330398-392-0/minicluster-data/ts-1/data/instance:
uuid: "b8fd7d2e8ee24183bfb08b183e95cde4"
format_stamp: "Formatted at 2026-06-20 04:25:56 on dist-test-slave-nk6d"
server_key: "a5062a7b217182b0402e785db370d642"
server_key_iv: "9a4a098480be871ef6a1fdabce8a3dad"
server_key_version: "encryptionkey@0"
I20260620 04:25:56.214856 9839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929496330398-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b8fd7d2e8ee24183bfb08b183e95cde4"
format_stamp: "Formatted at 2026-06-20 04:25:56 on dist-test-slave-nk6d"
server_key: "a5062a7b217182b0402e785db370d642"
server_key_iv: "9a4a098480be871ef6a1fdabce8a3dad"
server_key_version: "encryptionkey@0"
I20260620 04:25:56.218497 9839 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 04:25:56.220973 9853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:56.222214 9839 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:25:56.222321 9839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929496330398-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929496330398-392-0/minicluster-data/ts-1/wal
uuid: "b8fd7d2e8ee24183bfb08b183e95cde4"
format_stamp: "Formatted at 2026-06-20 04:25:56 on dist-test-slave-nk6d"
server_key: "a5062a7b217182b0402e785db370d642"
server_key_iv: "9a4a098480be871ef6a1fdabce8a3dad"
server_key_version: "encryptionkey@0"
I20260620 04:25:56.222419 9839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929496330398-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929496330398-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929496330398-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:25:56.246161 9839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:25:56.246874 9839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:25:56.247166 9839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:25:56.247812 9839 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:25:56.248900 9839 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:25:56.248950 9839 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:56.249002 9839 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:25:56.249025 9839 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:56.257385 9839 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34589
I20260620 04:25:56.257421 9966 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34589 every 8 connection(s)
I20260620 04:25:56.258262 9839 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929496330398-392-0/minicluster-data/ts-1/data/info.pb
I20260620 04:25:56.265283 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 9839
I20260620 04:25:56.265384 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929496330398-392-0/minicluster-data/ts-1/wal/instance
I20260620 04:25:56.265645 392 external_mini_cluster.cc:1428] Setting key 8f2c00510b5ba89a6a045277995afc68
I20260620 04:25:56.267174 9687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:56.260005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57023 (local address 127.0.98.62:41201)
0620 04:25:56.260136 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:56.260140 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:56.261149 (+ 1009us) server_negotiation.cc:407] Connection header received
0620 04:25:56.262500 (+ 1351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:56.262503 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:56.262560 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:56.262637 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:56.263532 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:56.264080 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:56.265251 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:56.265544 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:56.265634 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:56.266733 (+ 1099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:56.266737 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:56.266740 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:25:56.266752 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:56.266772 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:25:56.266779 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:56.266837 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:56.266973 (+ 136us) server_negotiation.cc:299] Negotiation successful
0620 04:25:56.267028 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 04:25:56.267602 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929496330398-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929496330398-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929496330398-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929496330398-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41201
--builtin_ntp_servers=127.0.98.20:39507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 04:25:56.268147 9969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:56.260363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41201 (local address 127.0.98.2:57023)
0620 04:25:56.260988 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:56.261026 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:56.262286 (+ 1260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:56.262816 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:56.262823 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:56.262907 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:56.263389 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:56.263401 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:56.264208 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:56.264212 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:56.265124 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:56.265135 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:56.265700 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:56.265705 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:56.265901 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:56.266479 (+ 578us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 04:25:56.266480 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:56.266519 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 04:25:56.266582 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:25:56.266849 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:25:56.266857 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:25:56.266863 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 04:25:56.266965 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 04:25:56.267321 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":111,"threads_started":1}
I20260620 04:25:56.269186 9967 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41201
I20260620 04:25:56.269392 9967 heartbeater.cc:464] Registering TS with master...
I20260620 04:25:56.269865 9967 heartbeater.cc:511] Master 127.0.98.62:41201 requested a full tablet report, sending...
I20260620 04:25:56.270795 9648 ts_manager.cc:194] Registered new tserver with Master: b8fd7d2e8ee24183bfb08b183e95cde4 (127.0.98.2:34589)
I20260620 04:25:56.271332 9648 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:57023
W20260620 04:25:56.376293 9970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:25:56.376549 9970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:25:56.376586 9970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:25:56.380592 9970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:25:56.380647 9970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:25:56.380702 9970 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 04:25:56.385557 9970 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929496330398-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929496330398-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929496330398-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41201
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9970
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929496330398-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:25:56.386605 9970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:25:56.387938 9970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:25:56.397017 9975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:25:56.397023 9978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:25:56.397030 9976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:25:56.397226 9970 server_base.cc:1034] running on GCE node
I20260620 04:25:56.397742 9970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:25:56.398348 9970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:25:56.399503 9970 hybrid_clock.cc:648] HybridClock initialized: now 1781929556399475 us; error 42 us; skew 500 ppm
I20260620 04:25:56.404693 9970 webserver.cc:460] Webserver started at http://127.0.98.3:45033/ using document root <none> and password file <none>
I20260620 04:25:56.405259 9970 fs_manager.cc:359] Metadata directory not provided
I20260620 04:25:56.405329 9970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:25:56.405517 9970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:25:56.407538 9970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929496330398-392-0/minicluster-data/ts-2/data/instance:
uuid: "3735dd0e04374ec093c73751879fa19d"
format_stamp: "Formatted at 2026-06-20 04:25:56 on dist-test-slave-nk6d"
server_key: "8d9097c653790092f9576cfa4fef33c5"
server_key_iv: "ee93f817397f1ba33c408cfe8f8868a2"
server_key_version: "encryptionkey@0"
I20260620 04:25:56.408093 9970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929496330398-392-0/minicluster-data/ts-2/wal/instance:
uuid: "3735dd0e04374ec093c73751879fa19d"
format_stamp: "Formatted at 2026-06-20 04:25:56 on dist-test-slave-nk6d"
server_key: "8d9097c653790092f9576cfa4fef33c5"
server_key_iv: "ee93f817397f1ba33c408cfe8f8868a2"
server_key_version: "encryptionkey@0"
I20260620 04:25:56.411604 9970 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:25:56.414243 9984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:56.415488 9970 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 04:25:56.415592 9970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929496330398-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929496330398-392-0/minicluster-data/ts-2/wal
uuid: "3735dd0e04374ec093c73751879fa19d"
format_stamp: "Formatted at 2026-06-20 04:25:56 on dist-test-slave-nk6d"
server_key: "8d9097c653790092f9576cfa4fef33c5"
server_key_iv: "ee93f817397f1ba33c408cfe8f8868a2"
server_key_version: "encryptionkey@0"
I20260620 04:25:56.415695 9970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929496330398-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929496330398-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929496330398-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:25:56.438297 9970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:25:56.438992 9970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:25:56.439186 9970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:25:56.439786 9970 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:25:56.440806 9970 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:25:56.440852 9970 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:56.440904 9970 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:25:56.440933 9970 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:56.449584 9970 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44271
I20260620 04:25:56.449618 10097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44271 every 8 connection(s)
I20260620 04:25:56.450505 9970 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929496330398-392-0/minicluster-data/ts-2/data/info.pb
I20260620 04:25:56.453553 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 9970
I20260620 04:25:56.453662 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929496330398-392-0/minicluster-data/ts-2/wal/instance
I20260620 04:25:56.453955 392 external_mini_cluster.cc:1428] Setting key a7babdec79532ab8d37d46d065c519ef
I20260620 04:25:56.460014 9687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:56.452369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60633 (local address 127.0.98.62:41201)
0620 04:25:56.452508 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:56.452512 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:56.453844 (+ 1332us) server_negotiation.cc:407] Connection header received
0620 04:25:56.455456 (+ 1612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:56.455459 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:56.455511 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:56.455582 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:56.456600 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:56.457103 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:56.458224 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:56.458438 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:56.458523 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:56.459536 (+ 1013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:56.459540 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:56.459541 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:25:56.459553 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:56.459573 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:25:56.459581 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:56.459636 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:56.459793 (+ 157us) server_negotiation.cc:299] Negotiation successful
0620 04:25:56.459851 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 04:25:56.460843 10100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:56.452918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41201 (local address 127.0.98.3:60633)
0620 04:25:56.453679 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:56.453733 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:56.455245 (+ 1512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:56.455784 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:56.455793 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:56.455905 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:56.456470 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:56.456484 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:56.457227 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:56.457231 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:56.458110 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:56.458119 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:56.458574 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:56.458578 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:56.458744 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:56.459292 (+ 548us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 04:25:56.459294 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:56.459334 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 04:25:56.459396 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:25:56.459666 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:25:56.459674 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:25:56.459680 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 04:25:56.459785 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 04:25:56.460088 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":100,"threads_started":1}
I20260620 04:25:56.461963 10098 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41201
I20260620 04:25:56.462196 10098 heartbeater.cc:464] Registering TS with master...
I20260620 04:25:56.462692 10098 heartbeater.cc:511] Master 127.0.98.62:41201 requested a full tablet report, sending...
I20260620 04:25:56.463723 9648 ts_manager.cc:194] Registered new tserver with Master: 3735dd0e04374ec093c73751879fa19d (127.0.98.3:44271)
I20260620 04:25:56.464296 9648 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:60633
I20260620 04:25:56.467687 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 04:25:56.470503 9687 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:25:56.469608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49148 (local address 127.0.98.62:41201)
0620 04:25:56.469725 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:56.469729 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:56.469892 (+ 163us) server_negotiation.cc:407] Connection header received
0620 04:25:56.470009 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:56.470011 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:56.470049 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:56.470131 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:56.470384 (+ 253us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49148: BlockingRecv error: recv got EOF from 127.0.0.1:49148 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 04:25:56.470765 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 9708
I20260620 04:25:56.475376 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 9839
I20260620 04:25:56.479612 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 9970
I20260620 04:25:56.483927 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 9617
2026-06-20T04:25:56Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (857 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-20T04:25:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:25:56Z Disabled control of system clock
I20260620 04:25:56.510236 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929496330398-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929496330398-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929496330398-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41659
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:41155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41659
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 04:25:56.617966 10109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:25:56.618232 10109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:25:56.618268 10109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:25:56.622273 10109 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:25:56.622340 10109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:25:56.622354 10109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:25:56.622370 10109 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:25:56.622383 10109 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:25:56.627184 10109 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929496330398-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929496330398-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41659
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41659
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10109
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929496330398-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:25:56.628299 10109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:25:56.629833 10109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:25:56.638078 10114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:25:56.638191 10109 server_base.cc:1034] running on GCE node
W20260620 04:25:56.638084 10115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:25:56.638087 10117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:25:56.638892 10109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:25:56.640000 10109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:25:56.641155 10109 hybrid_clock.cc:648] HybridClock initialized: now 1781929556641140 us; error 30 us; skew 500 ppm
I20260620 04:25:56.642897 10109 webserver.cc:460] Webserver started at http://127.0.98.62:38819/ using document root <none> and password file <none>
I20260620 04:25:56.643446 10109 fs_manager.cc:359] Metadata directory not provided
I20260620 04:25:56.643508 10109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:25:56.643690 10109 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:25:56.645673 10109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929496330398-392-0/minicluster-data/master-0/data/instance:
uuid: "030ea21de33b46c59e3187a05bb3fde1"
format_stamp: "Formatted at 2026-06-20 04:25:56 on dist-test-slave-nk6d"
server_key: "216e8717e9ac23da5bcb033dc1f15d59"
server_key_iv: "cc5067707b129def093f5579404ce316"
server_key_version: "encryptionkey@0"
I20260620 04:25:56.646201 10109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929496330398-392-0/minicluster-data/master-0/wal/instance:
uuid: "030ea21de33b46c59e3187a05bb3fde1"
format_stamp: "Formatted at 2026-06-20 04:25:56 on dist-test-slave-nk6d"
server_key: "216e8717e9ac23da5bcb033dc1f15d59"
server_key_iv: "cc5067707b129def093f5579404ce316"
server_key_version: "encryptionkey@0"
I20260620 04:25:56.649742 10109 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 04:25:56.652150 10123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:56.653355 10109 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:25:56.653465 10109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929496330398-392-0/minicluster-data/master-0/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929496330398-392-0/minicluster-data/master-0/wal
uuid: "030ea21de33b46c59e3187a05bb3fde1"
format_stamp: "Formatted at 2026-06-20 04:25:56 on dist-test-slave-nk6d"
server_key: "216e8717e9ac23da5bcb033dc1f15d59"
server_key_iv: "cc5067707b129def093f5579404ce316"
server_key_version: "encryptionkey@0"
I20260620 04:25:56.653564 10109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929496330398-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929496330398-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929496330398-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:25:56.657953 10109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:25:56.658522 10109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:25:56.658674 10109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:25:56.665254 10109 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41659
I20260620 04:25:56.665259 10175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41659 every 8 connection(s)
I20260620 04:25:56.666157 10109 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:25:56.669449 10176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:25:56.674824 10176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 030ea21de33b46c59e3187a05bb3fde1: Bootstrap starting.
I20260620 04:25:56.675411 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 10109
I20260620 04:25:56.675503 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929496330398-392-0/minicluster-data/master-0/wal/instance
I20260620 04:25:56.675768 392 external_mini_cluster.cc:1428] Setting key 0b44ad3dc38609f071e12917ebdb7773
I20260620 04:25:56.677491 10176 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 030ea21de33b46c59e3187a05bb3fde1: Neither blocks nor log segments found. Creating new log.
I20260620 04:25:56.678366 10176 log.cc:824] T 00000000000000000000000000000000 P 030ea21de33b46c59e3187a05bb3fde1: Log is configured to *not* fsync() on all Append() calls
I20260620 04:25:56.680588 10176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 030ea21de33b46c59e3187a05bb3fde1: No bootstrap required, opened a new log
I20260620 04:25:56.681453 10179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:56.677068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51272 (local address 127.0.98.62:41659)
0620 04:25:56.677743 (+ 675us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:56.677762 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:56.677804 (+ 42us) server_negotiation.cc:407] Connection header received
0620 04:25:56.678597 (+ 793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:56.678619 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:56.678969 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:56.679313 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:56.679977 (+ 664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:56.679982 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:56.679997 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:25:56.680008 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:56.680082 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:25:56.680105 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:56.680207 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:56.680308 (+ 101us) server_negotiation.cc:299] Negotiation successful
0620 04:25:56.680582 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":103,"threads_started":1}
I20260620 04:25:56.685171 10176 raft_consensus.cc:348] T 00000000000000000000000000000000 P 030ea21de33b46c59e3187a05bb3fde1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "030ea21de33b46c59e3187a05bb3fde1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41659 } }
I20260620 04:25:56.685454 10176 raft_consensus.cc:374] T 00000000000000000000000000000000 P 030ea21de33b46c59e3187a05bb3fde1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:25:56.685514 10176 raft_consensus.cc:729] T 00000000000000000000000000000000 P 030ea21de33b46c59e3187a05bb3fde1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 030ea21de33b46c59e3187a05bb3fde1, State: Initialized, Role: FOLLOWER
I20260620 04:25:56.685997 10176 consensus_queue.cc:260] T 00000000000000000000000000000000 P 030ea21de33b46c59e3187a05bb3fde1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "030ea21de33b46c59e3187a05bb3fde1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41659 } }
I20260620 04:25:56.686156 10176 raft_consensus.cc:388] T 00000000000000000000000000000000 P 030ea21de33b46c59e3187a05bb3fde1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:25:56.686260 10176 raft_consensus.cc:482] T 00000000000000000000000000000000 P 030ea21de33b46c59e3187a05bb3fde1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:25:56.686367 10176 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 030ea21de33b46c59e3187a05bb3fde1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:25:56.687501 10176 raft_consensus.cc:504] T 00000000000000000000000000000000 P 030ea21de33b46c59e3187a05bb3fde1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "030ea21de33b46c59e3187a05bb3fde1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41659 } }
I20260620 04:25:56.687891 10176 leader_election.cc:302] T 00000000000000000000000000000000 P 030ea21de33b46c59e3187a05bb3fde1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 030ea21de33b46c59e3187a05bb3fde1; no voters:
I20260620 04:25:56.688217 10176 leader_election.cc:288] T 00000000000000000000000000000000 P 030ea21de33b46c59e3187a05bb3fde1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:25:56.688305 10181 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 030ea21de33b46c59e3187a05bb3fde1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:25:56.688498 10181 raft_consensus.cc:686] T 00000000000000000000000000000000 P 030ea21de33b46c59e3187a05bb3fde1 [term 1 LEADER]: Becoming Leader. State: Replica: 030ea21de33b46c59e3187a05bb3fde1, State: Running, Role: LEADER
I20260620 04:25:56.688822 10181 consensus_queue.cc:237] T 00000000000000000000000000000000 P 030ea21de33b46c59e3187a05bb3fde1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "030ea21de33b46c59e3187a05bb3fde1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41659 } }
I20260620 04:25:56.689311 10176 sys_catalog.cc:564] T 00000000000000000000000000000000 P 030ea21de33b46c59e3187a05bb3fde1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:25:56.690308 10183 sys_catalog.cc:455] T 00000000000000000000000000000000 P 030ea21de33b46c59e3187a05bb3fde1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 030ea21de33b46c59e3187a05bb3fde1. Latest consensus state: current_term: 1 leader_uuid: "030ea21de33b46c59e3187a05bb3fde1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "030ea21de33b46c59e3187a05bb3fde1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41659 } } }
I20260620 04:25:56.690425 10183 sys_catalog.cc:458] T 00000000000000000000000000000000 P 030ea21de33b46c59e3187a05bb3fde1 [sys.catalog]: This master's current role is: LEADER
I20260620 04:25:56.690665 10182 sys_catalog.cc:455] T 00000000000000000000000000000000 P 030ea21de33b46c59e3187a05bb3fde1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "030ea21de33b46c59e3187a05bb3fde1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "030ea21de33b46c59e3187a05bb3fde1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41659 } } }
I20260620 04:25:56.690739 10182 sys_catalog.cc:458] T 00000000000000000000000000000000 P 030ea21de33b46c59e3187a05bb3fde1 [sys.catalog]: This master's current role is: LEADER
I20260620 04:25:56.691478 10190 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:25:56.693895 10190 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:25:56.698621 10190 catalog_manager.cc:1347] Generated new cluster ID: 7bb340aee3fc4e8ba2d44dddd9a38a2e
I20260620 04:25:56.698678 10190 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:25:56.704923 10190 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:25:56.705197 10190 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:25:56.710608 10190 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 030ea21de33b46c59e3187a05bb3fde1: Generated new TSK 0
I20260620 04:25:56.711236 10190 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 04:25:56.714174 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929496330398-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929496330398-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929496330398-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929496330398-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41659
--builtin_ntp_servers=127.0.98.20:41155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 04:25:56.822002 10200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:25:56.822239 10200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:25:56.822278 10200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:25:56.826244 10200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:25:56.826303 10200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:25:56.826357 10200 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 04:25:56.831091 10200 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929496330398-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929496330398-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929496330398-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41659
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10200
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929496330398-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:25:56.832099 10200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:25:56.833393 10200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:25:56.842525 10206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:25:56.842543 10205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:25:56.842734 10200 server_base.cc:1034] running on GCE node
W20260620 04:25:56.842535 10208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:25:56.843226 10200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:25:56.843827 10200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:25:56.844980 10200 hybrid_clock.cc:648] HybridClock initialized: now 1781929556844951 us; error 44 us; skew 500 ppm
I20260620 04:25:56.846706 10200 webserver.cc:460] Webserver started at http://127.0.98.1:38349/ using document root <none> and password file <none>
I20260620 04:25:56.847254 10200 fs_manager.cc:359] Metadata directory not provided
I20260620 04:25:56.847318 10200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:25:56.847504 10200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:25:56.849591 10200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929496330398-392-0/minicluster-data/ts-0/data/instance:
uuid: "d22662e84a674436993ac74c4d933d9e"
format_stamp: "Formatted at 2026-06-20 04:25:56 on dist-test-slave-nk6d"
server_key: "39bb2db74c0eda096c28d1715c879272"
server_key_iv: "38e639a241e67092bdf21bc482198eaa"
server_key_version: "encryptionkey@0"
I20260620 04:25:56.850157 10200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929496330398-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d22662e84a674436993ac74c4d933d9e"
format_stamp: "Formatted at 2026-06-20 04:25:56 on dist-test-slave-nk6d"
server_key: "39bb2db74c0eda096c28d1715c879272"
server_key_iv: "38e639a241e67092bdf21bc482198eaa"
server_key_version: "encryptionkey@0"
I20260620 04:25:56.853905 10200 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 04:25:56.856208 10214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:56.857450 10200 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:25:56.857547 10200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929496330398-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929496330398-392-0/minicluster-data/ts-0/wal
uuid: "d22662e84a674436993ac74c4d933d9e"
format_stamp: "Formatted at 2026-06-20 04:25:56 on dist-test-slave-nk6d"
server_key: "39bb2db74c0eda096c28d1715c879272"
server_key_iv: "38e639a241e67092bdf21bc482198eaa"
server_key_version: "encryptionkey@0"
I20260620 04:25:56.857636 10200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929496330398-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929496330398-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929496330398-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:25:56.862084 10200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:25:56.862663 10200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:25:56.862820 10200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:25:56.863339 10200 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:25:56.864259 10200 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:25:56.864305 10200 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:56.864355 10200 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:25:56.864383 10200 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:56.872517 10200 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43039
I20260620 04:25:56.872540 10327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43039 every 8 connection(s)
I20260620 04:25:56.873404 10200 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929496330398-392-0/minicluster-data/ts-0/data/info.pb
I20260620 04:25:56.879163 10179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:56.875137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34905 (local address 127.0.98.62:41659)
0620 04:25:56.875282 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:56.875286 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:56.876342 (+ 1056us) server_negotiation.cc:407] Connection header received
0620 04:25:56.877598 (+ 1256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:56.877602 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:56.877646 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:56.877726 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:56.878743 (+ 1017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:56.878747 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:56.878749 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:25:56.878762 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:56.878783 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:25:56.878790 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:56.878868 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:56.878992 (+ 124us) server_negotiation.cc:299] Negotiation successful
0620 04:25:56.879034 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 04:25:56.879941 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 10200
I20260620 04:25:56.880065 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929496330398-392-0/minicluster-data/ts-0/wal/instance
I20260620 04:25:56.880285 10330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:56.875519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41659 (local address 127.0.98.1:34905)
0620 04:25:56.876200 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:56.876240 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:56.877391 (+ 1151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:56.877904 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:56.877911 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:56.878001 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:56.878494 (+ 493us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 04:25:56.878495 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:56.878535 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 04:25:56.878601 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:25:56.878861 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:25:56.878869 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:25:56.878875 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 04:25:56.879026 (+ 151us) client_negotiation.cc:240] Negotiation successful
0620 04:25:56.879291 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":108,"threads_started":1}
I20260620 04:25:56.880367 392 external_mini_cluster.cc:1428] Setting key 1391079d6624f0234602fb5b76adb858
I20260620 04:25:56.881415 10328 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41659
I20260620 04:25:56.881647 10328 heartbeater.cc:464] Registering TS with master...
I20260620 04:25:56.882045 10328 heartbeater.cc:511] Master 127.0.98.62:41659 requested a full tablet report, sending...
I20260620 04:25:56.882335 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929496330398-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929496330398-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929496330398-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929496330398-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41659
--builtin_ntp_servers=127.0.98.20:41155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260620 04:25:56.883541 10139 ts_manager.cc:194] Registered new tserver with Master: d22662e84a674436993ac74c4d933d9e (127.0.98.1:43039)
W20260620 04:25:56.991926 10331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:25:56.992237 10331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:25:56.992296 10331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:25:56.996352 10331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:25:56.996404 10331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:25:56.996480 10331 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 04:25:57.001359 10331 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929496330398-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929496330398-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929496330398-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41659
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10331
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929496330398-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:25:57.002410 10331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:25:57.003830 10331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:25:57.012995 10336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:25:57.013067 10339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:25:57.013163 10331 server_base.cc:1034] running on GCE node
W20260620 04:25:57.012990 10337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:25:57.013589 10331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:25:57.014195 10331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:25:57.015358 10331 hybrid_clock.cc:648] HybridClock initialized: now 1781929557015337 us; error 30 us; skew 500 ppm
I20260620 04:25:57.017182 10331 webserver.cc:460] Webserver started at http://127.0.98.2:45743/ using document root <none> and password file <none>
I20260620 04:25:57.017740 10331 fs_manager.cc:359] Metadata directory not provided
I20260620 04:25:57.017804 10331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:25:57.017987 10331 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:25:57.019951 10331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929496330398-392-0/minicluster-data/ts-1/data/instance:
uuid: "b31263fbc1334a79b68f6d10786ff155"
format_stamp: "Formatted at 2026-06-20 04:25:57 on dist-test-slave-nk6d"
server_key: "d8b448a698d492b547e3cc7a3aef66d2"
server_key_iv: "56063a1122d55576f3723c5cdbc7695d"
server_key_version: "encryptionkey@0"
I20260620 04:25:57.020531 10331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929496330398-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b31263fbc1334a79b68f6d10786ff155"
format_stamp: "Formatted at 2026-06-20 04:25:57 on dist-test-slave-nk6d"
server_key: "d8b448a698d492b547e3cc7a3aef66d2"
server_key_iv: "56063a1122d55576f3723c5cdbc7695d"
server_key_version: "encryptionkey@0"
I20260620 04:25:57.024194 10331 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 04:25:57.026494 10345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:57.027688 10331 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:25:57.027798 10331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929496330398-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929496330398-392-0/minicluster-data/ts-1/wal
uuid: "b31263fbc1334a79b68f6d10786ff155"
format_stamp: "Formatted at 2026-06-20 04:25:57 on dist-test-slave-nk6d"
server_key: "d8b448a698d492b547e3cc7a3aef66d2"
server_key_iv: "56063a1122d55576f3723c5cdbc7695d"
server_key_version: "encryptionkey@0"
I20260620 04:25:57.027891 10331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929496330398-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929496330398-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929496330398-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:25:57.032228 10331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:25:57.032799 10331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:25:57.032946 10331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:25:57.033463 10331 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:25:57.034478 10331 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:25:57.034535 10331 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:57.034632 10331 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:25:57.034662 10331 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:57.042560 10331 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36633
I20260620 04:25:57.042606 10458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36633 every 8 connection(s)
I20260620 04:25:57.043463 10331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929496330398-392-0/minicluster-data/ts-1/data/info.pb
I20260620 04:25:57.048250 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 10331
I20260620 04:25:57.048367 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929496330398-392-0/minicluster-data/ts-1/wal/instance
I20260620 04:25:57.048619 392 external_mini_cluster.cc:1428] Setting key f29e628cb2feb89f6dc9e65010c54cf8
I20260620 04:25:57.049561 10179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:57.045261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54217 (local address 127.0.98.62:41659)
0620 04:25:57.045412 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:57.045416 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:57.046403 (+ 987us) server_negotiation.cc:407] Connection header received
0620 04:25:57.047734 (+ 1331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:57.047738 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:57.047778 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:57.047855 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:57.049104 (+ 1249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:57.049107 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:57.049111 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:25:57.049126 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:57.049151 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:25:57.049162 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:57.049235 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:57.049396 (+ 161us) server_negotiation.cc:299] Negotiation successful
0620 04:25:57.049437 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 04:25:57.050473 10461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:57.045563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41659 (local address 127.0.98.2:54217)
0620 04:25:57.046230 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:57.046272 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:57.047487 (+ 1215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:57.048107 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:57.048115 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:57.048217 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:57.048819 (+ 602us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 04:25:57.048821 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:57.048862 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 04:25:57.048939 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:25:57.049240 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:25:57.049248 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:25:57.049254 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 04:25:57.049422 (+ 168us) client_negotiation.cc:240] Negotiation successful
0620 04:25:57.049643 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":104,"threads_started":1}
I20260620 04:25:57.050547 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929496330398-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929496330398-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929496330398-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929496330398-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41659
--builtin_ntp_servers=127.0.98.20:41155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260620 04:25:57.051574 10459 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41659
I20260620 04:25:57.051823 10459 heartbeater.cc:464] Registering TS with master...
I20260620 04:25:57.052248 10459 heartbeater.cc:511] Master 127.0.98.62:41659 requested a full tablet report, sending...
I20260620 04:25:57.053278 10139 ts_manager.cc:194] Registered new tserver with Master: b31263fbc1334a79b68f6d10786ff155 (127.0.98.2:36633)
W20260620 04:25:57.157222 10462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:25:57.157498 10462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:25:57.157538 10462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:25:57.162065 10462 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:25:57.162127 10462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:25:57.162184 10462 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 04:25:57.166936 10462 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929496330398-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929496330398-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929496330398-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41659
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10462
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929496330398-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:25:57.167984 10462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:25:57.169324 10462 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 04:25:57.178483 10462 server_base.cc:1034] running on GCE node
W20260620 04:25:57.178463 10467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:25:57.178442 10470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:25:57.178447 10468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:25:57.179064 10462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:25:57.179690 10462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:25:57.180856 10462 hybrid_clock.cc:648] HybridClock initialized: now 1781929557180825 us; error 42 us; skew 500 ppm
I20260620 04:25:57.182593 10462 webserver.cc:460] Webserver started at http://127.0.98.3:45075/ using document root <none> and password file <none>
I20260620 04:25:57.183166 10462 fs_manager.cc:359] Metadata directory not provided
I20260620 04:25:57.183228 10462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:25:57.183415 10462 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:25:57.185477 10462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929496330398-392-0/minicluster-data/ts-2/data/instance:
uuid: "0e164acd05784b7f9ee5733c7bb19e71"
format_stamp: "Formatted at 2026-06-20 04:25:57 on dist-test-slave-nk6d"
server_key: "a366999115371c0174facf0703de7f3a"
server_key_iv: "d536e8c7024e75d97ddf2ff8cf6feecd"
server_key_version: "encryptionkey@0"
I20260620 04:25:57.186028 10462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929496330398-392-0/minicluster-data/ts-2/wal/instance:
uuid: "0e164acd05784b7f9ee5733c7bb19e71"
format_stamp: "Formatted at 2026-06-20 04:25:57 on dist-test-slave-nk6d"
server_key: "a366999115371c0174facf0703de7f3a"
server_key_iv: "d536e8c7024e75d97ddf2ff8cf6feecd"
server_key_version: "encryptionkey@0"
I20260620 04:25:57.190196 10462 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260620 04:25:57.192788 10476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:57.193957 10462 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260620 04:25:57.194052 10462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929496330398-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929496330398-392-0/minicluster-data/ts-2/wal
uuid: "0e164acd05784b7f9ee5733c7bb19e71"
format_stamp: "Formatted at 2026-06-20 04:25:57 on dist-test-slave-nk6d"
server_key: "a366999115371c0174facf0703de7f3a"
server_key_iv: "d536e8c7024e75d97ddf2ff8cf6feecd"
server_key_version: "encryptionkey@0"
I20260620 04:25:57.194177 10462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929496330398-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929496330398-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929496330398-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:25:57.198539 10462 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:25:57.199091 10462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:25:57.199242 10462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:25:57.199747 10462 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:25:57.200634 10462 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:25:57.200680 10462 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:57.200716 10462 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:25:57.200732 10462 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:57.208818 10462 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43941
I20260620 04:25:57.208866 10589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43941 every 8 connection(s)
I20260620 04:25:57.209702 10462 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929496330398-392-0/minicluster-data/ts-2/data/info.pb
I20260620 04:25:57.215396 10179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:57.211493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51737 (local address 127.0.98.62:41659)
0620 04:25:57.211628 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:57.211633 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:57.212734 (+ 1101us) server_negotiation.cc:407] Connection header received
0620 04:25:57.213942 (+ 1208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:57.213945 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:57.213980 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:57.214060 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:57.215037 (+ 977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:57.215040 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:57.215042 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:25:57.215053 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:57.215071 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:25:57.215079 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:57.215156 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:57.215263 (+ 107us) server_negotiation.cc:299] Negotiation successful
0620 04:25:57.215297 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 04:25:57.216275 10592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:57.211937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41659 (local address 127.0.98.3:51737)
0620 04:25:57.212591 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:57.212630 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:57.213737 (+ 1107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:57.214205 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:57.214211 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:57.214309 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:57.214801 (+ 492us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 04:25:57.214802 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:57.214840 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 04:25:57.214906 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:25:57.215148 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:25:57.215154 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:25:57.215157 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 04:25:57.215286 (+ 129us) client_negotiation.cc:240] Negotiation successful
0620 04:25:57.215476 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":108,"threads_started":1}
I20260620 04:25:57.216534 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 10462
I20260620 04:25:57.216627 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929496330398-392-0/minicluster-data/ts-2/wal/instance
I20260620 04:25:57.216885 392 external_mini_cluster.cc:1428] Setting key 894cb3bb3f1d362b5ed0e52d29f45510
I20260620 04:25:57.217396 10590 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41659
I20260620 04:25:57.217615 10590 heartbeater.cc:464] Registering TS with master...
I20260620 04:25:57.218097 10590 heartbeater.cc:511] Master 127.0.98.62:41659 requested a full tablet report, sending...
I20260620 04:25:57.219049 10139 ts_manager.cc:194] Registered new tserver with Master: 0e164acd05784b7f9ee5733c7bb19e71 (127.0.98.3:43941)
I20260620 04:25:57.230901 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 04:25:57.233798 10179 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:25:57.232877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51282 (local address 127.0.98.62:41659)
0620 04:25:57.233013 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:57.233017 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:57.233183 (+ 166us) server_negotiation.cc:407] Connection header received
0620 04:25:57.233328 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:57.233330 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:57.233368 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:57.233440 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:57.233665 (+ 225us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51282: BlockingRecv error: recv got EOF from 127.0.0.1:51282 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 04:25:57.233980 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 10200
I20260620 04:25:57.238891 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 10331
I20260620 04:25:57.243314 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 10462
I20260620 04:25:57.247567 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 10109
2026-06-20T04:25:57Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (763 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929496330398-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:25:57 dist-test-slave-nk6d krb5kdc[10600](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:25:57 dist-test-slave-nk6d krb5kdc[10600](info): set up 2 sockets
Jun 20 04:25:57 dist-test-slave-nk6d krb5kdc[10600](info): commencing operation
krb5kdc: starting...
W20260620 04:25:58.208781 392 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929496330398-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929496330398-392-0/krb5kdc/test-user.keytab.
Jun 20 04:25:58 dist-test-slave-nk6d krb5kdc[10600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929558, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:25:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:25:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 04:25:58.325294 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929496330398-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929496330398-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929496330398-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45297
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45297
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:25:58.432665 10613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:25:58.432945 10613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:25:58.432998 10613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:25:58.436954 10613 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:25:58.437009 10613 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:25:58.437036 10613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:25:58.437058 10613 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:25:58.437086 10613 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:25:58.442212 10613 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929496330398-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929496330398-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45297
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45297
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10613
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929496330398-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:25:58.443246 10613 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:25:58.444273 10613 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:25:58.452394 10618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:25:58.452461 10613 server_base.cc:1034] running on GCE node
W20260620 04:25:58.452376 10619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:25:58.452396 10621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:25:58.453132 10613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:25:58.454186 10613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:25:58.455351 10613 hybrid_clock.cc:648] HybridClock initialized: now 1781929558455334 us; error 28 us; skew 500 ppm
Jun 20 04:25:58 dist-test-slave-nk6d krb5kdc[10600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929558, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:25:58.458050 10613 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 04:25:58.458930 10613 webserver.cc:460] Webserver started at http://127.0.98.62:35129/ using document root <none> and password file <none>
I20260620 04:25:58.459458 10613 fs_manager.cc:359] Metadata directory not provided
I20260620 04:25:58.459522 10613 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:25:58.459704 10613 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:25:58.461769 10613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929496330398-392-0/minicluster-data/master-0/data/instance:
uuid: "2165971057284a7f9c283fa8f334f5b7"
format_stamp: "Formatted at 2026-06-20 04:25:58 on dist-test-slave-nk6d"
server_key: "d4e83af4f900b068d133a82dcfd11914"
server_key_iv: "46b9c0c45ef5a3cfee1a2092f39e5498"
server_key_version: "encryptionkey@0"
I20260620 04:25:58.462296 10613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929496330398-392-0/minicluster-data/master-0/wal/instance:
uuid: "2165971057284a7f9c283fa8f334f5b7"
format_stamp: "Formatted at 2026-06-20 04:25:58 on dist-test-slave-nk6d"
server_key: "d4e83af4f900b068d133a82dcfd11914"
server_key_iv: "46b9c0c45ef5a3cfee1a2092f39e5498"
server_key_version: "encryptionkey@0"
I20260620 04:25:58.465878 10613 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 04:25:58.468246 10628 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:58.469480 10613 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 04:25:58.469588 10613 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929496330398-392-0/minicluster-data/master-0/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929496330398-392-0/minicluster-data/master-0/wal
uuid: "2165971057284a7f9c283fa8f334f5b7"
format_stamp: "Formatted at 2026-06-20 04:25:58 on dist-test-slave-nk6d"
server_key: "d4e83af4f900b068d133a82dcfd11914"
server_key_iv: "46b9c0c45ef5a3cfee1a2092f39e5498"
server_key_version: "encryptionkey@0"
I20260620 04:25:58.469681 10613 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929496330398-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929496330398-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929496330398-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:25:58.504107 10613 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:25:58.507144 10613 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:25:58.507333 10613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:25:58.514014 10613 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45297
I20260620 04:25:58.514313 10680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45297 every 8 connection(s)
I20260620 04:25:58.514958 10613 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:25:58.517622 10681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:25:58.521545 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 10613
I20260620 04:25:58.521647 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929496330398-392-0/minicluster-data/master-0/wal/instance
I20260620 04:25:58.521915 392 external_mini_cluster.cc:1428] Setting key fec210ded32a9a42fb198207e5fb333e
I20260620 04:25:58.523279 10681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2165971057284a7f9c283fa8f334f5b7: Bootstrap starting.
I20260620 04:25:58.525882 10681 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2165971057284a7f9c283fa8f334f5b7: Neither blocks nor log segments found. Creating new log.
I20260620 04:25:58.526726 10681 log.cc:824] T 00000000000000000000000000000000 P 2165971057284a7f9c283fa8f334f5b7: Log is configured to *not* fsync() on all Append() calls
I20260620 04:25:58.528759 10681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2165971057284a7f9c283fa8f334f5b7: No bootstrap required, opened a new log
Jun 20 04:25:58 dist-test-slave-nk6d krb5kdc[10600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929558, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:25:58.532819 10681 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2165971057284a7f9c283fa8f334f5b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2165971057284a7f9c283fa8f334f5b7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45297 } }
I20260620 04:25:58.533102 10681 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2165971057284a7f9c283fa8f334f5b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:25:58.533157 10681 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2165971057284a7f9c283fa8f334f5b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2165971057284a7f9c283fa8f334f5b7, State: Initialized, Role: FOLLOWER
I20260620 04:25:58.533771 10681 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2165971057284a7f9c283fa8f334f5b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2165971057284a7f9c283fa8f334f5b7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45297 } }
I20260620 04:25:58.533926 10681 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2165971057284a7f9c283fa8f334f5b7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:25:58.533989 10681 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2165971057284a7f9c283fa8f334f5b7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:25:58.534082 10681 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2165971057284a7f9c283fa8f334f5b7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:25:58.535116 10681 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2165971057284a7f9c283fa8f334f5b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2165971057284a7f9c283fa8f334f5b7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45297 } }
I20260620 04:25:58.535488 10681 leader_election.cc:302] T 00000000000000000000000000000000 P 2165971057284a7f9c283fa8f334f5b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2165971057284a7f9c283fa8f334f5b7; no voters:
I20260620 04:25:58.535830 10681 leader_election.cc:288] T 00000000000000000000000000000000 P 2165971057284a7f9c283fa8f334f5b7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:25:58.536037 10686 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2165971057284a7f9c283fa8f334f5b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:25:58.536211 10686 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2165971057284a7f9c283fa8f334f5b7 [term 1 LEADER]: Becoming Leader. State: Replica: 2165971057284a7f9c283fa8f334f5b7, State: Running, Role: LEADER
I20260620 04:25:58.536511 10686 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2165971057284a7f9c283fa8f334f5b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2165971057284a7f9c283fa8f334f5b7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45297 } }
I20260620 04:25:58.536922 10684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:58.523386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33662 (local address 127.0.98.62:45297)
0620 04:25:58.523815 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:58.523833 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:58.523875 (+ 42us) server_negotiation.cc:407] Connection header received
0620 04:25:58.524692 (+ 817us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:58.524714 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:58.525119 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:58.525470 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:58.526128 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:58.527443 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:58.528370 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:58.529036 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:58.529189 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:58.531725 (+ 2536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:58.531732 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:58.531748 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:25:58.531793 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:58.533752 (+ 1959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:58.534269 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:58.534275 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:58.534281 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:58.534416 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:58.534658 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:58.534660 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:58.534662 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:58.535187 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:58.535377 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:58.535660 (+ 283us) server_negotiation.cc:299] Negotiation successful
0620 04:25:58.535994 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":119,"threads_started":1}
I20260620 04:25:58.537106 10681 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2165971057284a7f9c283fa8f334f5b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:25:58.538196 10688 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2165971057284a7f9c283fa8f334f5b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2165971057284a7f9c283fa8f334f5b7. Latest consensus state: current_term: 1 leader_uuid: "2165971057284a7f9c283fa8f334f5b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2165971057284a7f9c283fa8f334f5b7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45297 } } }
I20260620 04:25:58.538203 10687 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2165971057284a7f9c283fa8f334f5b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2165971057284a7f9c283fa8f334f5b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2165971057284a7f9c283fa8f334f5b7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45297 } } }
I20260620 04:25:58.538324 10688 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2165971057284a7f9c283fa8f334f5b7 [sys.catalog]: This master's current role is: LEADER
I20260620 04:25:58.538324 10687 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2165971057284a7f9c283fa8f334f5b7 [sys.catalog]: This master's current role is: LEADER
I20260620 04:25:58.539153 10695 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:25:58.542274 10695 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:25:58.547819 10695 catalog_manager.cc:1347] Generated new cluster ID: 9fb261ce377547e39c738f7d564668a9
I20260620 04:25:58.547886 10695 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:25:58.556489 10695 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:25:58.557530 10695 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:25:58.565237 10695 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2165971057284a7f9c283fa8f334f5b7: Generated new TSK 0
I20260620 04:25:58.565963 10695 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929496330398-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929496330398-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 04:25:58.600524 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929496330398-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929496330398-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929496330398-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929496330398-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45297
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929496330398-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:25:58.732764 10707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:25:58.733031 10707 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:25:58.733072 10707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:25:58.737442 10707 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:25:58.737517 10707 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:25:58.737582 10707 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 04:25:58.743103 10707 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929496330398-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929496330398-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929496330398-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929496330398-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45297
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10707
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929496330398-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:25:58.744123 10707 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:25:58.745138 10707 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:25:58.754762 10712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:25:58.754762 10713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:25:58.754873 10707 server_base.cc:1034] running on GCE node
W20260620 04:25:58.754801 10715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:25:58.755374 10707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:25:58.756049 10707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:25:58.757246 10707 hybrid_clock.cc:648] HybridClock initialized: now 1781929558757211 us; error 31 us; skew 500 ppm
Jun 20 04:25:58 dist-test-slave-nk6d krb5kdc[10600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929558, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:25:58.760236 10707 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 04:25:58.761090 10707 webserver.cc:460] Webserver started at http://127.0.98.1:42077/ using document root <none> and password file <none>
I20260620 04:25:58.761595 10707 fs_manager.cc:359] Metadata directory not provided
I20260620 04:25:58.761649 10707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:25:58.761824 10707 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:25:58.763761 10707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929496330398-392-0/minicluster-data/ts-0/data/instance:
uuid: "355b8dd3bb82497793fd8769cef7da36"
format_stamp: "Formatted at 2026-06-20 04:25:58 on dist-test-slave-nk6d"
server_key: "b166d89fcd1c8470beeccedbf28dcd2d"
server_key_iv: "6e6119e9f668d1649dead584eda4a643"
server_key_version: "encryptionkey@0"
I20260620 04:25:58.764300 10707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929496330398-392-0/minicluster-data/ts-0/wal/instance:
uuid: "355b8dd3bb82497793fd8769cef7da36"
format_stamp: "Formatted at 2026-06-20 04:25:58 on dist-test-slave-nk6d"
server_key: "b166d89fcd1c8470beeccedbf28dcd2d"
server_key_iv: "6e6119e9f668d1649dead584eda4a643"
server_key_version: "encryptionkey@0"
I20260620 04:25:58.768046 10707 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 04:25:58.770437 10722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:58.771749 10707 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:25:58.771889 10707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929496330398-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929496330398-392-0/minicluster-data/ts-0/wal
uuid: "355b8dd3bb82497793fd8769cef7da36"
format_stamp: "Formatted at 2026-06-20 04:25:58 on dist-test-slave-nk6d"
server_key: "b166d89fcd1c8470beeccedbf28dcd2d"
server_key_iv: "6e6119e9f668d1649dead584eda4a643"
server_key_version: "encryptionkey@0"
I20260620 04:25:58.772030 10707 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929496330398-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929496330398-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929496330398-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:25:58.792915 10707 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:25:58.795627 10707 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:25:58.795773 10707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:25:58.796331 10707 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:25:58.797093 10707 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:25:58.797143 10707 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:58.797183 10707 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:25:58.797202 10707 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:58.805325 10707 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38273
I20260620 04:25:58.805356 10835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38273 every 8 connection(s)
I20260620 04:25:58.806205 10707 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929496330398-392-0/minicluster-data/ts-0/data/info.pb
I20260620 04:25:58.807368 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 10707
I20260620 04:25:58.807477 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929496330398-392-0/minicluster-data/ts-0/wal/instance
I20260620 04:25:58.807783 392 external_mini_cluster.cc:1428] Setting key 9b4cf2b5e736ae5a94c6e4f1d8a7e707
Jun 20 04:25:58 dist-test-slave-nk6d krb5kdc[10600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929558, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 04:25:58.820248 10684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:58.807976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35445 (local address 127.0.98.62:45297)
0620 04:25:58.808086 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:58.808090 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:58.809003 (+ 913us) server_negotiation.cc:407] Connection header received
0620 04:25:58.810505 (+ 1502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:58.810507 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:58.810552 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:58.810622 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:58.811839 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:58.812424 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:58.814140 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:58.814369 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:58.814443 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:58.817032 (+ 2589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:58.817035 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:58.817037 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:25:58.817059 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:58.818283 (+ 1224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:58.818742 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:58.818745 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:58.818746 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:58.818818 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:58.819129 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:58.819132 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:58.819134 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:58.819302 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:58.819381 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:58.820019 (+ 638us) server_negotiation.cc:299] Negotiation successful
0620 04:25:58.820124 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 04:25:58.821205 10838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:58.808414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45297 (local address 127.0.98.1:35445)
0620 04:25:58.808858 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:58.808899 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:58.810264 (+ 1365us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:58.810793 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:58.810801 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:58.811224 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:58.811696 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:58.811709 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:58.812540 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:58.812543 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:58.814005 (+ 1462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:58.814016 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:58.814518 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:58.814522 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:58.814723 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:58.815282 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:58.815300 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:58.816869 (+ 1569us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:25:58.818397 (+ 1528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:58.818402 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:58.818417 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:58.818647 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:58.818909 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:58.818913 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:58.818915 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:58.819042 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:58.819422 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:25:58.819428 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:25:58.819693 (+ 265us) client_negotiation.cc:769] Sending connection context
0620 04:25:58.819894 (+ 201us) client_negotiation.cc:240] Negotiation successful
0620 04:25:58.820244 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":124,"threads_started":1}
I20260620 04:25:58.822378 10836 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45297
I20260620 04:25:58.822633 10836 heartbeater.cc:464] Registering TS with master...
I20260620 04:25:58.823215 10836 heartbeater.cc:511] Master 127.0.98.62:45297 requested a full tablet report, sending...
I20260620 04:25:58.824869 10645 ts_manager.cc:194] Registered new tserver with Master: 355b8dd3bb82497793fd8769cef7da36 (127.0.98.1:38273)
I20260620 04:25:58.826004 10645 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:35445
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929496330398-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929496330398-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 04:25:58.835199 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929496330398-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929496330398-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929496330398-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929496330398-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45297
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929496330398-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:25:58.942833 10841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:25:58.943096 10841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:25:58.943137 10841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:25:58.947636 10841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:25:58.947710 10841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:25:58.947801 10841 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 04:25:58.953632 10841 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929496330398-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929496330398-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929496330398-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929496330398-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45297
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10841
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929496330398-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:25:58.954646 10841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:25:58.955622 10841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:25:58.965252 10847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:25:58.965265 10849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:25:58.965337 10841 server_base.cc:1034] running on GCE node
W20260620 04:25:58.965261 10846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:25:58.965912 10841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:25:58.966557 10841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:25:58.967788 10841 hybrid_clock.cc:648] HybridClock initialized: now 1781929558967764 us; error 31 us; skew 500 ppm
Jun 20 04:25:58 dist-test-slave-nk6d krb5kdc[10600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929558, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:25:58.970692 10841 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 04:25:58.971655 10841 webserver.cc:460] Webserver started at http://127.0.98.2:39505/ using document root <none> and password file <none>
I20260620 04:25:58.972260 10841 fs_manager.cc:359] Metadata directory not provided
I20260620 04:25:58.972337 10841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:25:58.972555 10841 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:25:58.974689 10841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929496330398-392-0/minicluster-data/ts-1/data/instance:
uuid: "4efdc32282844d22af1093fc8122066e"
format_stamp: "Formatted at 2026-06-20 04:25:58 on dist-test-slave-nk6d"
server_key: "8527857e3c0c6b96d07864a9721e6bbf"
server_key_iv: "7a8a1620c6bcc7526ea9ae9ff602f279"
server_key_version: "encryptionkey@0"
I20260620 04:25:58.975345 10841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929496330398-392-0/minicluster-data/ts-1/wal/instance:
uuid: "4efdc32282844d22af1093fc8122066e"
format_stamp: "Formatted at 2026-06-20 04:25:58 on dist-test-slave-nk6d"
server_key: "8527857e3c0c6b96d07864a9721e6bbf"
server_key_iv: "7a8a1620c6bcc7526ea9ae9ff602f279"
server_key_version: "encryptionkey@0"
I20260620 04:25:58.979341 10841 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 04:25:58.981845 10856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:58.983107 10841 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:25:58.983213 10841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929496330398-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929496330398-392-0/minicluster-data/ts-1/wal
uuid: "4efdc32282844d22af1093fc8122066e"
format_stamp: "Formatted at 2026-06-20 04:25:58 on dist-test-slave-nk6d"
server_key: "8527857e3c0c6b96d07864a9721e6bbf"
server_key_iv: "7a8a1620c6bcc7526ea9ae9ff602f279"
server_key_version: "encryptionkey@0"
I20260620 04:25:58.983310 10841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929496330398-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929496330398-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929496330398-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:25:59.005414 10841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:25:59.008435 10841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:25:59.008613 10841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:25:59.009200 10841 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:25:59.010084 10841 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:25:59.010130 10841 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:59.010183 10841 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:25:59.010210 10841 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:59.018115 10841 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38521
I20260620 04:25:59.018143 10969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38521 every 8 connection(s)
I20260620 04:25:59.018965 10841 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929496330398-392-0/minicluster-data/ts-1/data/info.pb
I20260620 04:25:59.021410 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 10841
I20260620 04:25:59.021510 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929496330398-392-0/minicluster-data/ts-1/wal/instance
I20260620 04:25:59.021797 392 external_mini_cluster.cc:1428] Setting key af0daf54162641bcfa524e8358344195
Jun 20 04:25:59 dist-test-slave-nk6d krb5kdc[10600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929558, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 04:25:59.032790 10684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:59.020784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52441 (local address 127.0.98.62:45297)
0620 04:25:59.020974 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:59.020978 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:59.021798 (+ 820us) server_negotiation.cc:407] Connection header received
0620 04:25:59.023138 (+ 1340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:59.023142 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:59.023193 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:59.023274 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:59.024803 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:59.025349 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:59.026500 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:59.026715 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:59.026800 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:59.029392 (+ 2592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:59.029395 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:59.029397 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:25:59.029421 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:59.030815 (+ 1394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:59.031315 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:59.031318 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:59.031319 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:59.031390 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:59.031700 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:59.031703 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:59.031705 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:59.031881 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:59.031985 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:59.032509 (+ 524us) server_negotiation.cc:299] Negotiation successful
0620 04:25:59.032626 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260620 04:25:59.033728 10972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:59.021165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45297 (local address 127.0.98.2:52441)
0620 04:25:59.021630 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:59.021681 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:59.022909 (+ 1228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:59.023469 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:59.023478 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:59.023948 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:59.024644 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:59.024661 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:59.025483 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:59.025486 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:59.026380 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:59.026390 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:59.026850 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:59.026854 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:59.027015 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:59.027565 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:59.027581 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:59.029225 (+ 1644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:25:59.030919 (+ 1694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:59.030924 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:59.030948 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:59.031185 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:59.031477 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:59.031480 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:59.031482 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:59.031613 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:59.032005 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:25:59.032011 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:25:59.032250 (+ 239us) client_negotiation.cc:769] Sending connection context
0620 04:25:59.032429 (+ 179us) client_negotiation.cc:240] Negotiation successful
0620 04:25:59.032750 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":114,"threads_started":1}
I20260620 04:25:59.034974 10970 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45297
I20260620 04:25:59.035211 10970 heartbeater.cc:464] Registering TS with master...
I20260620 04:25:59.035756 10970 heartbeater.cc:511] Master 127.0.98.62:45297 requested a full tablet report, sending...
I20260620 04:25:59.036868 10645 ts_manager.cc:194] Registered new tserver with Master: 4efdc32282844d22af1093fc8122066e (127.0.98.2:38521)
I20260620 04:25:59.037451 10645 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:52441
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929496330398-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929496330398-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 04:25:59.049113 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929496330398-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929496330398-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929496330398-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929496330398-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45297
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929496330398-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:25:59.158335 10975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:25:59.158607 10975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:25:59.158653 10975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:25:59.162783 10975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:25:59.162846 10975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:25:59.162914 10975 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 04:25:59.168258 10975 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929496330398-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929496330398-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929496330398-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929496330398-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45297
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10975
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929496330398-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:25:59.169313 10975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:25:59.170313 10975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 04:25:59.179927 10975 server_base.cc:1034] running on GCE node
W20260620 04:25:59.179930 10981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:25:59.180145 10983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:25:59.179930 10980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:25:59.180541 10975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:25:59.181149 10975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:25:59.182301 10975 hybrid_clock.cc:648] HybridClock initialized: now 1781929559182284 us; error 29 us; skew 500 ppm
Jun 20 04:25:59 dist-test-slave-nk6d krb5kdc[10600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929559, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:25:59.185139 10975 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 04:25:59.186022 10975 webserver.cc:460] Webserver started at http://127.0.98.3:35905/ using document root <none> and password file <none>
I20260620 04:25:59.186568 10975 fs_manager.cc:359] Metadata directory not provided
I20260620 04:25:59.186633 10975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:25:59.186820 10975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:25:59.188832 10975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929496330398-392-0/minicluster-data/ts-2/data/instance:
uuid: "d2eb793f5ea7470389b77f911feada18"
format_stamp: "Formatted at 2026-06-20 04:25:59 on dist-test-slave-nk6d"
server_key: "8d910f805ad3efa8c3a763ce2ad9e83c"
server_key_iv: "21f730ad76f1b62e01949ea4fa725fba"
server_key_version: "encryptionkey@0"
I20260620 04:25:59.189327 10975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929496330398-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d2eb793f5ea7470389b77f911feada18"
format_stamp: "Formatted at 2026-06-20 04:25:59 on dist-test-slave-nk6d"
server_key: "8d910f805ad3efa8c3a763ce2ad9e83c"
server_key_iv: "21f730ad76f1b62e01949ea4fa725fba"
server_key_version: "encryptionkey@0"
I20260620 04:25:59.192885 10975 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 04:25:59.195242 10990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:59.196517 10975 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:25:59.196635 10975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929496330398-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929496330398-392-0/minicluster-data/ts-2/wal
uuid: "d2eb793f5ea7470389b77f911feada18"
format_stamp: "Formatted at 2026-06-20 04:25:59 on dist-test-slave-nk6d"
server_key: "8d910f805ad3efa8c3a763ce2ad9e83c"
server_key_iv: "21f730ad76f1b62e01949ea4fa725fba"
server_key_version: "encryptionkey@0"
I20260620 04:25:59.196728 10975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929496330398-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929496330398-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929496330398-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:25:59.225464 10975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:25:59.228336 10975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:25:59.228523 10975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:25:59.229110 10975 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:25:59.229928 10975 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:25:59.229975 10975 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:59.230027 10975 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:25:59.230055 10975 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:25:59.238679 10975 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:32931
I20260620 04:25:59.238710 11103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:32931 every 8 connection(s)
I20260620 04:25:59.239624 10975 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929496330398-392-0/minicluster-data/ts-2/data/info.pb
I20260620 04:25:59.245462 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 10975
I20260620 04:25:59.245568 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929496330398-392-0/minicluster-data/ts-2/wal/instance
I20260620 04:25:59.245831 392 external_mini_cluster.cc:1428] Setting key a7bb25aa70f9c582e98d49e400f3c216
Jun 20 04:25:59 dist-test-slave-nk6d krb5kdc[10600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929559, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:25:59.252954 10684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:59.241475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59645 (local address 127.0.98.62:45297)
0620 04:25:59.241620 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:59.241624 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:59.242552 (+ 928us) server_negotiation.cc:407] Connection header received
0620 04:25:59.243800 (+ 1248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:59.243804 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:59.243861 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:59.243936 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:59.245182 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:59.245748 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:59.247048 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:59.247256 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:59.247340 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:59.249663 (+ 2323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:59.249666 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:59.249669 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:25:59.249691 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:59.251017 (+ 1326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:59.251483 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:59.251487 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:59.251488 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:59.251561 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:59.251866 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:59.251870 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:59.251871 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:59.252101 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:59.252187 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:59.252674 (+ 487us) server_negotiation.cc:299] Negotiation successful
0620 04:25:59.252794 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 04:25:59.253749 11106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:59.241882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45297 (local address 127.0.98.3:59645)
0620 04:25:59.242364 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:59.242408 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:59.243581 (+ 1173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:59.244161 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:59.244168 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:59.244539 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:25:59.245030 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:59.245044 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:59.245867 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:59.245870 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:59.246940 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:59.246950 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:59.247388 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:59.247393 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:59.247557 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:59.248120 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:25:59.248137 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:25:59.249512 (+ 1375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:25:59.251127 (+ 1615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:59.251133 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:59.251148 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:59.251387 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:59.251644 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:25:59.251647 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:25:59.251649 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:25:59.251779 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:25:59.252196 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:25:59.252203 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:25:59.252443 (+ 240us) client_negotiation.cc:769] Sending connection context
0620 04:25:59.252616 (+ 173us) client_negotiation.cc:240] Negotiation successful
0620 04:25:59.252932 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":117,"threads_started":1}
I20260620 04:25:59.254906 11104 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45297
I20260620 04:25:59.255123 11104 heartbeater.cc:464] Registering TS with master...
I20260620 04:25:59.255594 11104 heartbeater.cc:511] Master 127.0.98.62:45297 requested a full tablet report, sending...
I20260620 04:25:59.256619 10645 ts_manager.cc:194] Registered new tserver with Master: d2eb793f5ea7470389b77f911feada18 (127.0.98.3:32931)
I20260620 04:25:59.257228 10645 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59645
I20260620 04:25:59.259538 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 04:25:59.268536 10684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:59.261376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33672 (local address 127.0.98.62:45297)
0620 04:25:59.261486 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:59.261489 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:59.261670 (+ 181us) server_negotiation.cc:407] Connection header received
0620 04:25:59.261732 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:59.261734 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:59.261783 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:59.261857 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:25:59.262461 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:59.262963 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:59.263809 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:59.264094 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:59.264184 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:59.265369 (+ 1185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:25:59.265372 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:25:59.265374 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:25:59.265395 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:25:59.266730 (+ 1335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:59.267164 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:59.267167 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:59.267168 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:59.267248 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:25:59.267583 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:25:59.267587 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:25:59.267588 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:25:59.267806 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:25:59.267931 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:59.268303 (+ 372us) server_negotiation.cc:299] Negotiation successful
0620 04:25:59.268415 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 04:25:59.271607 10645 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33672:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 04:25:59.274648 10645 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 04:25:59.289644 11120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:59.283575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38521 (local address 127.0.0.1:37562)
0620 04:25:59.284846 (+ 1271us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:59.284858 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:59.284945 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:59.285606 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:59.285609 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:59.285636 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:25:59.285739 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:59.285745 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:59.286823 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:59.286826 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:59.288218 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:59.288228 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:59.289127 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:59.289130 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:59.289230 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:59.289264 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 04:25:59.289372 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 04:25:59.289491 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1214,"mutex_wait_us":3,"spinlock_wait_cycles":896,"thread_start_us":1154,"threads_started":1}
I20260620 04:25:59.289652 11119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:59.284122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37562 (local address 127.0.98.2:38521)
0620 04:25:59.284679 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:59.284690 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:59.284957 (+ 267us) server_negotiation.cc:407] Connection header received
0620 04:25:59.285044 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:59.285050 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:59.285278 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:59.285443 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:25:59.285859 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:59.286708 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:59.288327 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:59.289006 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:59.289101 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:59.289251 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:59.289408 (+ 157us) server_negotiation.cc:299] Negotiation successful
0620 04:25:59.289501 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":169,"threads_started":1}
I20260620 04:25:59.289971 11115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:59.283384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38273 (local address 127.0.0.1:45910)
0620 04:25:59.285141 (+ 1757us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:59.285153 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:59.285289 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:59.285895 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:59.285898 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:59.285915 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:25:59.285983 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:59.285988 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:59.287328 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:59.287331 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:59.288531 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:59.288541 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:59.289658 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:59.289661 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:59.289735 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:59.289756 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 04:25:59.289814 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 04:25:59.289872 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1703,"mutex_wait_us":665,"thread_start_us":92,"threads_started":1}
I20260620 04:25:59.289851 11118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:59.283379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:32931 (local address 127.0.0.1:37556)
0620 04:25:59.284506 (+ 1127us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:59.284570 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:59.284698 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:59.285566 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:59.285572 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:59.285593 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:25:59.285730 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:59.285736 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:59.287154 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:59.287157 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:59.288295 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:59.288302 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:59.289455 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:59.289459 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:59.289543 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:59.289564 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 04:25:59.289628 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 04:25:59.289697 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":946,"mutex_wait_us":2,"spinlock_wait_cycles":1280,"thread_start_us":81,"threads_started":1}
I20260620 04:25:59.290170 11116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:59.283519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37556 (local address 127.0.98.3:32931)
0620 04:25:59.284592 (+ 1073us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:59.284604 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:59.284636 (+ 32us) server_negotiation.cc:407] Connection header received
0620 04:25:59.284950 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:59.284957 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:59.285406 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:59.285558 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:25:59.285845 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:59.287022 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:59.288418 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:59.289149 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:59.289254 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:59.289391 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:59.289945 (+ 554us) server_negotiation.cc:299] Negotiation successful
0620 04:25:59.290039 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":977,"thread_start_us":114,"threads_started":1}
I20260620 04:25:59.290498 11117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:59.283616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45910 (local address 127.0.98.1:38273)
0620 04:25:59.284398 (+ 782us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:59.284413 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:59.285228 (+ 815us) server_negotiation.cc:407] Connection header received
0620 04:25:59.285430 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:59.285435 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:59.285597 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:59.285712 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:25:59.286098 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:59.287212 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:59.288655 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:59.289392 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:59.289485 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:59.289606 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:59.290263 (+ 657us) server_negotiation.cc:299] Negotiation successful
0620 04:25:59.290366 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":666,"thread_start_us":60,"threads_started":1}
I20260620 04:25:59.293011 10904 tablet_service.cc:1467] Processing CreateTablet for tablet 2b33ec72aa6043babaca393b3d4bde47 (DEFAULT_TABLE table=test-table [id=ec06fa9adf68452f8e71f822cba6bb8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:25:59.293085 11038 tablet_service.cc:1467] Processing CreateTablet for tablet 2b33ec72aa6043babaca393b3d4bde47 (DEFAULT_TABLE table=test-table [id=ec06fa9adf68452f8e71f822cba6bb8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:25:59.293085 10770 tablet_service.cc:1467] Processing CreateTablet for tablet 2b33ec72aa6043babaca393b3d4bde47 (DEFAULT_TABLE table=test-table [id=ec06fa9adf68452f8e71f822cba6bb8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:25:59.294008 10904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b33ec72aa6043babaca393b3d4bde47. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:25:59.294092 10770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b33ec72aa6043babaca393b3d4bde47. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:25:59.294099 11038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b33ec72aa6043babaca393b3d4bde47. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:25:59.300469 11121 tablet_bootstrap.cc:492] T 2b33ec72aa6043babaca393b3d4bde47 P 4efdc32282844d22af1093fc8122066e: Bootstrap starting.
I20260620 04:25:59.301826 11123 tablet_bootstrap.cc:492] T 2b33ec72aa6043babaca393b3d4bde47 P d2eb793f5ea7470389b77f911feada18: Bootstrap starting.
I20260620 04:25:59.301826 11122 tablet_bootstrap.cc:492] T 2b33ec72aa6043babaca393b3d4bde47 P 355b8dd3bb82497793fd8769cef7da36: Bootstrap starting.
I20260620 04:25:59.302904 11121 tablet_bootstrap.cc:654] T 2b33ec72aa6043babaca393b3d4bde47 P 4efdc32282844d22af1093fc8122066e: Neither blocks nor log segments found. Creating new log.
I20260620 04:25:59.303601 11121 log.cc:824] T 2b33ec72aa6043babaca393b3d4bde47 P 4efdc32282844d22af1093fc8122066e: Log is configured to *not* fsync() on all Append() calls
I20260620 04:25:59.303983 11122 tablet_bootstrap.cc:654] T 2b33ec72aa6043babaca393b3d4bde47 P 355b8dd3bb82497793fd8769cef7da36: Neither blocks nor log segments found. Creating new log.
I20260620 04:25:59.303984 11123 tablet_bootstrap.cc:654] T 2b33ec72aa6043babaca393b3d4bde47 P d2eb793f5ea7470389b77f911feada18: Neither blocks nor log segments found. Creating new log.
I20260620 04:25:59.305167 11122 log.cc:824] T 2b33ec72aa6043babaca393b3d4bde47 P 355b8dd3bb82497793fd8769cef7da36: Log is configured to *not* fsync() on all Append() calls
I20260620 04:25:59.305166 11123 log.cc:824] T 2b33ec72aa6043babaca393b3d4bde47 P d2eb793f5ea7470389b77f911feada18: Log is configured to *not* fsync() on all Append() calls
I20260620 04:25:59.305428 11121 tablet_bootstrap.cc:492] T 2b33ec72aa6043babaca393b3d4bde47 P 4efdc32282844d22af1093fc8122066e: No bootstrap required, opened a new log
I20260620 04:25:59.305606 11121 ts_tablet_manager.cc:1397] T 2b33ec72aa6043babaca393b3d4bde47 P 4efdc32282844d22af1093fc8122066e: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260620 04:25:59.307092 11122 tablet_bootstrap.cc:492] T 2b33ec72aa6043babaca393b3d4bde47 P 355b8dd3bb82497793fd8769cef7da36: No bootstrap required, opened a new log
I20260620 04:25:59.307112 11123 tablet_bootstrap.cc:492] T 2b33ec72aa6043babaca393b3d4bde47 P d2eb793f5ea7470389b77f911feada18: No bootstrap required, opened a new log
I20260620 04:25:59.307281 11122 ts_tablet_manager.cc:1397] T 2b33ec72aa6043babaca393b3d4bde47 P 355b8dd3bb82497793fd8769cef7da36: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 04:25:59.307281 11123 ts_tablet_manager.cc:1397] T 2b33ec72aa6043babaca393b3d4bde47 P d2eb793f5ea7470389b77f911feada18: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 04:25:59.308647 11121 raft_consensus.cc:348] T 2b33ec72aa6043babaca393b3d4bde47 P 4efdc32282844d22af1093fc8122066e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2eb793f5ea7470389b77f911feada18" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32931 } } peers { permanent_uuid: "355b8dd3bb82497793fd8769cef7da36" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38273 } } peers { permanent_uuid: "4efdc32282844d22af1093fc8122066e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38521 } }
I20260620 04:25:59.308884 11121 raft_consensus.cc:374] T 2b33ec72aa6043babaca393b3d4bde47 P 4efdc32282844d22af1093fc8122066e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:25:59.308935 11121 raft_consensus.cc:729] T 2b33ec72aa6043babaca393b3d4bde47 P 4efdc32282844d22af1093fc8122066e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4efdc32282844d22af1093fc8122066e, State: Initialized, Role: FOLLOWER
I20260620 04:25:59.309350 11121 consensus_queue.cc:260] T 2b33ec72aa6043babaca393b3d4bde47 P 4efdc32282844d22af1093fc8122066e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2eb793f5ea7470389b77f911feada18" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32931 } } peers { permanent_uuid: "355b8dd3bb82497793fd8769cef7da36" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38273 } } peers { permanent_uuid: "4efdc32282844d22af1093fc8122066e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38521 } }
I20260620 04:25:59.309959 10970 heartbeater.cc:503] Master 127.0.98.62:45297 was elected leader, sending a full tablet report...
I20260620 04:25:59.311297 11121 ts_tablet_manager.cc:1428] T 2b33ec72aa6043babaca393b3d4bde47 P 4efdc32282844d22af1093fc8122066e: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260620 04:25:59.311127 11123 raft_consensus.cc:348] T 2b33ec72aa6043babaca393b3d4bde47 P d2eb793f5ea7470389b77f911feada18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2eb793f5ea7470389b77f911feada18" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32931 } } peers { permanent_uuid: "355b8dd3bb82497793fd8769cef7da36" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38273 } } peers { permanent_uuid: "4efdc32282844d22af1093fc8122066e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38521 } }
I20260620 04:25:59.311126 11122 raft_consensus.cc:348] T 2b33ec72aa6043babaca393b3d4bde47 P 355b8dd3bb82497793fd8769cef7da36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2eb793f5ea7470389b77f911feada18" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32931 } } peers { permanent_uuid: "355b8dd3bb82497793fd8769cef7da36" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38273 } } peers { permanent_uuid: "4efdc32282844d22af1093fc8122066e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38521 } }
I20260620 04:25:59.311388 11123 raft_consensus.cc:374] T 2b33ec72aa6043babaca393b3d4bde47 P d2eb793f5ea7470389b77f911feada18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:25:59.311388 11122 raft_consensus.cc:374] T 2b33ec72aa6043babaca393b3d4bde47 P 355b8dd3bb82497793fd8769cef7da36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:25:59.311444 11122 raft_consensus.cc:729] T 2b33ec72aa6043babaca393b3d4bde47 P 355b8dd3bb82497793fd8769cef7da36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 355b8dd3bb82497793fd8769cef7da36, State: Initialized, Role: FOLLOWER
I20260620 04:25:59.311445 11123 raft_consensus.cc:729] T 2b33ec72aa6043babaca393b3d4bde47 P d2eb793f5ea7470389b77f911feada18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2eb793f5ea7470389b77f911feada18, State: Initialized, Role: FOLLOWER
I20260620 04:25:59.311892 11123 consensus_queue.cc:260] T 2b33ec72aa6043babaca393b3d4bde47 P d2eb793f5ea7470389b77f911feada18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2eb793f5ea7470389b77f911feada18" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32931 } } peers { permanent_uuid: "355b8dd3bb82497793fd8769cef7da36" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38273 } } peers { permanent_uuid: "4efdc32282844d22af1093fc8122066e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38521 } }
I20260620 04:25:59.311892 11122 consensus_queue.cc:260] T 2b33ec72aa6043babaca393b3d4bde47 P 355b8dd3bb82497793fd8769cef7da36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2eb793f5ea7470389b77f911feada18" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32931 } } peers { permanent_uuid: "355b8dd3bb82497793fd8769cef7da36" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38273 } } peers { permanent_uuid: "4efdc32282844d22af1093fc8122066e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38521 } }
I20260620 04:25:59.312922 11122 ts_tablet_manager.cc:1428] T 2b33ec72aa6043babaca393b3d4bde47 P 355b8dd3bb82497793fd8769cef7da36: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260620 04:25:59.312922 11123 ts_tablet_manager.cc:1428] T 2b33ec72aa6043babaca393b3d4bde47 P d2eb793f5ea7470389b77f911feada18: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260620 04:25:59.313267 10836 heartbeater.cc:503] Master 127.0.98.62:45297 was elected leader, sending a full tablet report...
I20260620 04:25:59.313267 11104 heartbeater.cc:503] Master 127.0.98.62:45297 was elected leader, sending a full tablet report...
I20260620 04:25:59.353212 11129 raft_consensus.cc:482] T 2b33ec72aa6043babaca393b3d4bde47 P d2eb793f5ea7470389b77f911feada18 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:25:59.353433 11129 raft_consensus.cc:504] T 2b33ec72aa6043babaca393b3d4bde47 P d2eb793f5ea7470389b77f911feada18 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2eb793f5ea7470389b77f911feada18" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32931 } } peers { permanent_uuid: "355b8dd3bb82497793fd8769cef7da36" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38273 } } peers { permanent_uuid: "4efdc32282844d22af1093fc8122066e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38521 } }
I20260620 04:25:59.354624 11129 leader_election.cc:288] T 2b33ec72aa6043babaca393b3d4bde47 P d2eb793f5ea7470389b77f911feada18 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 355b8dd3bb82497793fd8769cef7da36 (127.0.98.1:38273), 4efdc32282844d22af1093fc8122066e (127.0.98.2:38521)
I20260620 04:25:59.358127 11117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:59.354930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46995 (local address 127.0.98.1:38273)
0620 04:25:59.355076 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:59.355081 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:59.355100 (+ 19us) server_negotiation.cc:407] Connection header received
0620 04:25:59.355266 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:59.355269 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:59.355312 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:59.355377 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:25:59.355640 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:59.356192 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:59.357198 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:59.357605 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:59.357700 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:59.357762 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:59.357937 (+ 175us) server_negotiation.cc:299] Negotiation successful
0620 04:25:59.358002 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 04:25:59.358146 11106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:59.354841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38273 (local address 127.0.98.3:46995)
0620 04:25:59.355024 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:59.355038 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:59.355134 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:59.355401 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:59.355404 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:59.355430 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:25:59.355535 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:59.355540 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:59.356296 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:59.356299 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:59.357074 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:59.357083 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:59.357723 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:59.357726 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:59.357825 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:59.357845 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 04:25:59.357909 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 04:25:59.357978 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"mutex_wait_us":4,"spinlock_wait_cycles":1408}
I20260620 04:25:59.358541 11119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:59.355043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40973 (local address 127.0.98.2:38521)
0620 04:25:59.355150 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:59.355153 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:59.355375 (+ 222us) server_negotiation.cc:407] Connection header received
0620 04:25:59.355522 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:59.355526 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:59.355568 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:59.355635 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:25:59.355919 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:59.356533 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:59.357586 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:59.357996 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:59.358088 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:59.358154 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:59.358349 (+ 195us) server_negotiation.cc:299] Negotiation successful
0620 04:25:59.358426 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 04:25:59.358862 11130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:59.354950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38521 (local address 127.0.98.3:40973)
0620 04:25:59.355282 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:59.355292 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 04:25:59.355422 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:25:59.355655 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:25:59.355658 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:25:59.355673 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:25:59.355777 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:59.355782 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:59.356652 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:59.356655 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:59.357462 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:25:59.357471 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:25:59.358106 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:25:59.358109 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:25:59.358208 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:59.358230 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 04:25:59.358673 (+ 443us) client_negotiation.cc:240] Negotiation successful
0620 04:25:59.358742 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"mutex_wait_us":2,"thread_start_us":146,"threads_started":1}
I20260620 04:25:59.358873 10790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b33ec72aa6043babaca393b3d4bde47" candidate_uuid: "d2eb793f5ea7470389b77f911feada18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "355b8dd3bb82497793fd8769cef7da36" is_pre_election: true
I20260620 04:25:59.359177 10790 raft_consensus.cc:2454] T 2b33ec72aa6043babaca393b3d4bde47 P 355b8dd3bb82497793fd8769cef7da36 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d2eb793f5ea7470389b77f911feada18 in term 0.
I20260620 04:25:59.359475 10924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b33ec72aa6043babaca393b3d4bde47" candidate_uuid: "d2eb793f5ea7470389b77f911feada18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4efdc32282844d22af1093fc8122066e" is_pre_election: true
I20260620 04:25:59.359705 10994 leader_election.cc:302] T 2b33ec72aa6043babaca393b3d4bde47 P d2eb793f5ea7470389b77f911feada18 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 355b8dd3bb82497793fd8769cef7da36, d2eb793f5ea7470389b77f911feada18; no voters:
I20260620 04:25:59.359757 10924 raft_consensus.cc:2454] T 2b33ec72aa6043babaca393b3d4bde47 P 4efdc32282844d22af1093fc8122066e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d2eb793f5ea7470389b77f911feada18 in term 0.
I20260620 04:25:59.360049 11129 raft_consensus.cc:2781] T 2b33ec72aa6043babaca393b3d4bde47 P d2eb793f5ea7470389b77f911feada18 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 04:25:59.360138 11129 raft_consensus.cc:482] T 2b33ec72aa6043babaca393b3d4bde47 P d2eb793f5ea7470389b77f911feada18 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 04:25:59.360185 11129 raft_consensus.cc:3037] T 2b33ec72aa6043babaca393b3d4bde47 P d2eb793f5ea7470389b77f911feada18 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:25:59.361207 11129 raft_consensus.cc:504] T 2b33ec72aa6043babaca393b3d4bde47 P d2eb793f5ea7470389b77f911feada18 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2eb793f5ea7470389b77f911feada18" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32931 } } peers { permanent_uuid: "355b8dd3bb82497793fd8769cef7da36" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38273 } } peers { permanent_uuid: "4efdc32282844d22af1093fc8122066e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38521 } }
I20260620 04:25:59.361524 11129 leader_election.cc:288] T 2b33ec72aa6043babaca393b3d4bde47 P d2eb793f5ea7470389b77f911feada18 [CANDIDATE]: Term 1 election: Requested vote from peers 355b8dd3bb82497793fd8769cef7da36 (127.0.98.1:38273), 4efdc32282844d22af1093fc8122066e (127.0.98.2:38521)
I20260620 04:25:59.361843 10924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b33ec72aa6043babaca393b3d4bde47" candidate_uuid: "d2eb793f5ea7470389b77f911feada18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4efdc32282844d22af1093fc8122066e"
I20260620 04:25:59.361918 10790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b33ec72aa6043babaca393b3d4bde47" candidate_uuid: "d2eb793f5ea7470389b77f911feada18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "355b8dd3bb82497793fd8769cef7da36"
I20260620 04:25:59.361964 10924 raft_consensus.cc:3037] T 2b33ec72aa6043babaca393b3d4bde47 P 4efdc32282844d22af1093fc8122066e [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:25:59.362057 10790 raft_consensus.cc:3037] T 2b33ec72aa6043babaca393b3d4bde47 P 355b8dd3bb82497793fd8769cef7da36 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:25:59.363178 10924 raft_consensus.cc:2454] T 2b33ec72aa6043babaca393b3d4bde47 P 4efdc32282844d22af1093fc8122066e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2eb793f5ea7470389b77f911feada18 in term 1.
I20260620 04:25:59.363178 10790 raft_consensus.cc:2454] T 2b33ec72aa6043babaca393b3d4bde47 P 355b8dd3bb82497793fd8769cef7da36 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2eb793f5ea7470389b77f911feada18 in term 1.
I20260620 04:25:59.363459 10992 leader_election.cc:302] T 2b33ec72aa6043babaca393b3d4bde47 P d2eb793f5ea7470389b77f911feada18 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4efdc32282844d22af1093fc8122066e, d2eb793f5ea7470389b77f911feada18; no voters:
I20260620 04:25:59.363622 11129 raft_consensus.cc:2781] T 2b33ec72aa6043babaca393b3d4bde47 P d2eb793f5ea7470389b77f911feada18 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:25:59.363847 11129 raft_consensus.cc:686] T 2b33ec72aa6043babaca393b3d4bde47 P d2eb793f5ea7470389b77f911feada18 [term 1 LEADER]: Becoming Leader. State: Replica: d2eb793f5ea7470389b77f911feada18, State: Running, Role: LEADER
I20260620 04:25:59.364214 11129 consensus_queue.cc:237] T 2b33ec72aa6043babaca393b3d4bde47 P d2eb793f5ea7470389b77f911feada18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2eb793f5ea7470389b77f911feada18" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32931 } } peers { permanent_uuid: "355b8dd3bb82497793fd8769cef7da36" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38273 } } peers { permanent_uuid: "4efdc32282844d22af1093fc8122066e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38521 } }
I20260620 04:25:59.367404 10645 catalog_manager.cc:5495] T 2b33ec72aa6043babaca393b3d4bde47 P d2eb793f5ea7470389b77f911feada18 reported cstate change: term changed from 0 to 1, leader changed from <none> to d2eb793f5ea7470389b77f911feada18 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "d2eb793f5ea7470389b77f911feada18" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2eb793f5ea7470389b77f911feada18" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32931 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "355b8dd3bb82497793fd8769cef7da36" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38273 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4efdc32282844d22af1093fc8122066e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38521 } health_report { overall_health: UNKNOWN } } }
I20260620 04:25:59.387809 11116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:59.384214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37570 (local address 127.0.98.3:32931)
0620 04:25:59.384342 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 04:25:59.384346 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:25:59.384360 (+ 14us) server_negotiation.cc:407] Connection header received
0620 04:25:59.384462 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:25:59.384464 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:25:59.384539 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:25:59.384626 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:25:59.384806 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:59.385302 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:59.386211 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:25:59.386461 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:25:59.386553 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:25:59.386963 (+ 410us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:25:59.387385 (+ 422us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:25:59.387485 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 04:25:59.387595 (+ 110us) server_negotiation.cc:299] Negotiation successful
0620 04:25:59.387662 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 04:25:59.393806 10924 raft_consensus.cc:1261] T 2b33ec72aa6043babaca393b3d4bde47 P 4efdc32282844d22af1093fc8122066e [term 1 FOLLOWER]: Refusing update from remote peer d2eb793f5ea7470389b77f911feada18: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:25:59.393806 10790 raft_consensus.cc:1261] T 2b33ec72aa6043babaca393b3d4bde47 P 355b8dd3bb82497793fd8769cef7da36 [term 1 FOLLOWER]: Refusing update from remote peer d2eb793f5ea7470389b77f911feada18: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:25:59.394654 11131 consensus_queue.cc:1035] T 2b33ec72aa6043babaca393b3d4bde47 P d2eb793f5ea7470389b77f911feada18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4efdc32282844d22af1093fc8122066e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:25:59.394865 11129 consensus_queue.cc:1035] T 2b33ec72aa6043babaca393b3d4bde47 P d2eb793f5ea7470389b77f911feada18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "355b8dd3bb82497793fd8769cef7da36" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:25:59.403337 11134 mvcc.cc:205] Tried to move back new op lower bound from 7298783475269578752 to 7298783475158302720. Current Snapshot: MvccSnapshot[applied={T|T < 7298783475269578752}]
I20260620 04:25:59.403646 11136 mvcc.cc:205] Tried to move back new op lower bound from 7298783475269578752 to 7298783475158302720. Current Snapshot: MvccSnapshot[applied={T|T < 7298783475269578752}]
I20260620 04:25:59.406504 11135 mvcc.cc:205] Tried to move back new op lower bound from 7298783475269578752 to 7298783475158302720. Current Snapshot: MvccSnapshot[applied={T|T < 7298783475269578752}]
I20260620 04:25:59.412241 10644 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33672:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:25:59.412395 10644 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33672:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:25:59.414750 10644 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2165971057284a7f9c283fa8f334f5b7: Sending DeleteTablet for 3 replicas of tablet 2b33ec72aa6043babaca393b3d4bde47
I20260620 04:25:59.415508 11038 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b33ec72aa6043babaca393b3d4bde47 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:25:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37556
I20260620 04:25:59.415531 10770 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b33ec72aa6043babaca393b3d4bde47 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:25:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45910
I20260620 04:25:59.415620 10904 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b33ec72aa6043babaca393b3d4bde47 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:25:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37562
I20260620 04:25:59.416098 11144 tablet_replica.cc:330] T 2b33ec72aa6043babaca393b3d4bde47 P d2eb793f5ea7470389b77f911feada18: stopping tablet replica
I20260620 04:25:59.416370 11143 tablet_replica.cc:330] T 2b33ec72aa6043babaca393b3d4bde47 P 355b8dd3bb82497793fd8769cef7da36: stopping tablet replica
I20260620 04:25:59.416407 11144 raft_consensus.cc:2229] T 2b33ec72aa6043babaca393b3d4bde47 P d2eb793f5ea7470389b77f911feada18 [term 1 LEADER]: Raft consensus shutting down.
I20260620 04:25:59.416651 11145 tablet_replica.cc:330] T 2b33ec72aa6043babaca393b3d4bde47 P 4efdc32282844d22af1093fc8122066e: stopping tablet replica
I20260620 04:25:59.416741 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 10707
I20260620 04:25:59.416759 11144 raft_consensus.cc:2258] T 2b33ec72aa6043babaca393b3d4bde47 P d2eb793f5ea7470389b77f911feada18 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:25:59.417135 11143 raft_consensus.cc:2229] T 2b33ec72aa6043babaca393b3d4bde47 P 355b8dd3bb82497793fd8769cef7da36 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:25:59.417496 11145 raft_consensus.cc:2229] T 2b33ec72aa6043babaca393b3d4bde47 P 4efdc32282844d22af1093fc8122066e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:25:59.417932 11145 raft_consensus.cc:2258] T 2b33ec72aa6043babaca393b3d4bde47 P 4efdc32282844d22af1093fc8122066e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:25:59.419636 11145 ts_tablet_manager.cc:1905] T 2b33ec72aa6043babaca393b3d4bde47 P 4efdc32282844d22af1093fc8122066e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:25:59.420207 11144 ts_tablet_manager.cc:1905] T 2b33ec72aa6043babaca393b3d4bde47 P d2eb793f5ea7470389b77f911feada18: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:25:59.422497 11145 ts_tablet_manager.cc:1918] T 2b33ec72aa6043babaca393b3d4bde47 P 4efdc32282844d22af1093fc8122066e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:25:59.422683 11145 log.cc:1196] T 2b33ec72aa6043babaca393b3d4bde47 P 4efdc32282844d22af1093fc8122066e: Deleting WAL directory at /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929496330398-392-0/minicluster-data/ts-1/wal/wals/2b33ec72aa6043babaca393b3d4bde47
I20260620 04:25:59.422694 11144 ts_tablet_manager.cc:1918] T 2b33ec72aa6043babaca393b3d4bde47 P d2eb793f5ea7470389b77f911feada18: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:25:59.422777 11144 log.cc:1196] T 2b33ec72aa6043babaca393b3d4bde47 P d2eb793f5ea7470389b77f911feada18: Deleting WAL directory at /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929496330398-392-0/minicluster-data/ts-2/wal/wals/2b33ec72aa6043babaca393b3d4bde47
I20260620 04:25:59.423131 11145 ts_tablet_manager.cc:1939] T 2b33ec72aa6043babaca393b3d4bde47 P 4efdc32282844d22af1093fc8122066e: Deleting consensus metadata
I20260620 04:25:59.423301 11144 ts_tablet_manager.cc:1939] T 2b33ec72aa6043babaca393b3d4bde47 P d2eb793f5ea7470389b77f911feada18: Deleting consensus metadata
W20260620 04:25:59.423297 10632 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38273 (error 108)
I20260620 04:25:59.424482 10629 catalog_manager.cc:4841] TS d2eb793f5ea7470389b77f911feada18 (127.0.98.3:32931): tablet 2b33ec72aa6043babaca393b3d4bde47 (table test-table [id=ec06fa9adf68452f8e71f822cba6bb8b]) successfully deleted
I20260620 04:25:59.424494 10630 catalog_manager.cc:4841] TS 4efdc32282844d22af1093fc8122066e (127.0.98.2:38521): tablet 2b33ec72aa6043babaca393b3d4bde47 (table test-table [id=ec06fa9adf68452f8e71f822cba6bb8b]) successfully deleted
I20260620 04:25:59.425209 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 10841
I20260620 04:25:59.425438 11115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:25:59.425174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38273 (local address 127.0.0.1:45924)
0620 04:25:59.425299 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 04:25:59.425363 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 04:25:59.426975 10632 catalog_manager.cc:4584] TS 355b8dd3bb82497793fd8769cef7da36 (127.0.98.1:38273): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2b33ec72aa6043babaca393b3d4bde47: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38273: connect: Connection refused (error 111)
I20260620 04:25:59.430703 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 10975
I20260620 04:25:59.435461 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 10613
2026-06-20T04:25:59Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2188 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:25:59 dist-test-slave-nk6d krb5kdc[11149](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:25:59 dist-test-slave-nk6d krb5kdc[11149](info): set up 2 sockets
Jun 20 04:25:59 dist-test-slave-nk6d krb5kdc[11149](info): commencing operation
krb5kdc: starting...
W20260620 04:26:00.392664 392 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/krb5kdc/test-user.keytab.
Jun 20 04:26:00 dist-test-slave-nk6d krb5kdc[11149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929560, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:26:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:26:00Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260620 04:26:00.546178 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taske80qNG/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/minicluster-data/postgres -l logfile start
2026-06-20 04:26:02.972 UTC [11179] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 04:26:02.973 UTC [11179] LOG: listening on IPv4 address "127.0.98.20", port 47543
2026-06-20 04:26:02.976 UTC [11179] LOG: listening on Unix socket "/tmp/.s.PGSQL.47543"
2026-06-20 04:26:03.002 UTC [11182] LOG: database system was shut down at 2026-06-20 04:26:01 UTC
2026-06-20 04:26:03.005 UTC [11179] LOG: database system is ready to accept connections
I20260620 04:26:03.868793 392 mini_postgres.cc:96] Postgres bound to 47543
2026-06-20 04:26:03.875 UTC [11190] FATAL: database "slave" does not exist
127.0.98.20:47543 - accepting connections
I20260620 04:26:03.875623 392 mini_ranger.cc:161] Starting Ranger...
I20260620 04:26:03.889708 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 04:26:04.294296 392 mini_ranger.cc:87] Created ranger Postgres database
I20260620 04:26:04.294386 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/minicluster-data/ranger-admin
2026-06-20 04:26:04,557 [I] DB FLAVOR :POSTGRES
2026-06-20 04:26:04,557 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 04:26:04,557 [I] Checking connection..
2026-06-20 04:26:04,557 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske80qNG/build/debug/bin/postgresql.jar:/tmp/dist-test-taske80qNG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:47543/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 04:26:04,799 [I] Checking connection passed.
2026-06-20 04:26:04,800 [I] --------- Verifying version history table ---------
2026-06-20 04:26:04,800 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske80qNG/build/debug/bin/postgresql.jar:/tmp/dist-test-taske80qNG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:47543/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 04:26:05,051 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 04:26:05,051 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske80qNG/build/debug/bin/postgresql.jar:/tmp/dist-test-taske80qNG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:47543/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 04:26:05,274 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 04:26:05,274 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 04:26:05,274 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske80qNG/build/debug/bin/postgresql.jar:/tmp/dist-test-taske80qNG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:47543/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taske80qNG/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 04:26:05.520 UTC [11254] WARNING: there is no transaction in progress
2026-06-20 04:26:05,525 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 04:26:05,525 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske80qNG/build/debug/bin/postgresql.jar:/tmp/dist-test-taske80qNG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:47543/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 04:26:05,755 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 04:26:05,755 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 04:26:05,755 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske80qNG/build/debug/bin/postgresql.jar:/tmp/dist-test-taske80qNG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:47543/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 04:26:06,010 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske80qNG/build/debug/bin/postgresql.jar:/tmp/dist-test-taske80qNG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:47543/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 04:26:06,239 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske80qNG/build/debug/bin/postgresql.jar:/tmp/dist-test-taske80qNG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:47543/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-nk6d.c.gcp-upstream.internal','N') ;"
2026-06-20 04:26:06,456 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske80qNG/build/debug/bin/postgresql.jar:/tmp/dist-test-taske80qNG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:47543/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 04:26:06,682 [I] Table x_portal_user does not exist in database ranger
2026-06-20 04:26:06,682 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske80qNG/build/debug/bin/postgresql.jar:/tmp/dist-test-taske80qNG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:47543/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 04:26:06,906 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 04:26:06,906 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske80qNG/build/debug/bin/postgresql.jar:/tmp/dist-test-taske80qNG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:47543/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 04:26:07,149 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske80qNG/build/debug/bin/postgresql.jar:/tmp/dist-test-taske80qNG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:47543/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 04:26:07,401 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 04:26:07,401 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske80qNG/build/debug/bin/postgresql.jar:/tmp/dist-test-taske80qNG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:47543/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taske80qNG/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 04:26:08.333 UTC [11362] WARNING: there is no transaction in progress
2026-06-20 04:26:08.348 UTC [11362] WARNING: there is no transaction in progress
2026-06-20 04:26:08.363 UTC [11362] WARNING: there is no transaction in progress
2026-06-20 04:26:08.378 UTC [11362] WARNING: there is no transaction in progress
2026-06-20 04:26:08.393 UTC [11362] WARNING: there is no transaction in progress
2026-06-20 04:26:08.412 UTC [11362] WARNING: there is no transaction in progress
2026-06-20 04:26:08.469 UTC [11362] WARNING: there is no transaction in progress
2026-06-20 04:26:08.475 UTC [11362] WARNING: there is no transaction in progress
2026-06-20 04:26:08.489 UTC [11362] WARNING: there is no transaction in progress
2026-06-20 04:26:08.499 UTC [11362] WARNING: there is no transaction in progress
2026-06-20 04:26:08.509 UTC [11362] WARNING: there is no transaction in progress
2026-06-20 04:26:08.522 UTC [11362] WARNING: there is no transaction in progress
2026-06-20 04:26:08.531 UTC [11362] WARNING: there is no transaction in progress
2026-06-20 04:26:08.537 UTC [11362] WARNING: there is no transaction in progress
2026-06-20 04:26:08.543 UTC [11362] WARNING: there is no transaction in progress
2026-06-20 04:26:08.549 UTC [11362] WARNING: there is no transaction in progress
2026-06-20 04:26:09,147 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 04:26:09,148 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske80qNG/build/debug/bin/postgresql.jar:/tmp/dist-test-taske80qNG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:47543/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 04:26:09,381 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 04:26:09,382 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske80qNG/build/debug/bin/postgresql.jar:/tmp/dist-test-taske80qNG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:47543/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 04:26:09,623 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 04:26:09,623 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske80qNG/build/debug/bin/postgresql.jar:/tmp/dist-test-taske80qNG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:47543/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 04:26:09,871 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 04:26:09,872 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske80qNG/build/debug/bin/postgresql.jar:/tmp/dist-test-taske80qNG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:47543/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 04:26:10,121 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske80qNG/build/debug/bin/postgresql.jar:/tmp/dist-test-taske80qNG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:47543/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 04:26:10,372 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske80qNG/build/debug/bin/postgresql.jar:/tmp/dist-test-taske80qNG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:47543/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-nk6d.c.gcp-upstream.internal';"
2026-06-20 04:26:10,586 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 04:26:10,587 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske80qNG/build/debug/bin/postgresql.jar:/tmp/dist-test-taske80qNG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:47543/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 04:26:10,810 [I] DB_PATCHES have already been applied
I20260620 04:26:10.815477 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taske80qNG/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taske80qNG/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taske80qNG/build/debug/bin/postgresql.jar:/tmp/dist-test-taske80qNG/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 04:26:10.815538 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260620 04:26:10.818656 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:58279
Jun 20, 2026 4:26:11 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 4:26:11 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 4:26:11 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 4:26:11 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 4:26:11 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 4:26:11 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 4:26:11 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 04:26:11.830014 392 mini_ranger.cc:160] Time spent starting Ranger: real 7.954s user 0.002s sys 0.007s
Jun 20, 2026 4:26:30 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 4:26:30 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Jun 20, 2026 4:26:30 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 4:26:30 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 4:26:30 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 4:26:30 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 4:26:30 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 4:26:30 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 4:26:30 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 4:26:30 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 4:26:30 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 4:26:30 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 4:26:30 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 4:26:30 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 4:26:30 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 4:26:30 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 4:26:30 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 4:26:31 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 04:26:32.066440 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 04:26:32 dist-test-slave-nk6d krb5kdc[11149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929592, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 04:26:32.147893 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 04:26:32.157564 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 04:26:32.908365 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 04:26:32.908445 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/minicluster-data/ranger-kms
I20260620 04:26:32.908514 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:47543
2026-06-20 04:26:33,071 [I] DB FLAVOR :POSTGRES
2026-06-20 04:26:33,071 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 04:26:33,071 [I] Checking connection
2026-06-20 04:26:33,071 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske80qNG/build/debug/bin/postgresql.jar:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:47543/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 04:26:33,311 [I] connection success
2026-06-20 04:26:33,311 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 04:26:33,311 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 04:26:33,311 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske80qNG/build/debug/bin/postgresql.jar:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:47543/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 04:26:33,541 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 04:26:33,542 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 04:26:33,542 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 04:26:33,542 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske80qNG/build/debug/bin/postgresql.jar:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:47543/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 04:26:33,788 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 04:26:34.220851 392 mini_ranger_kms.cc:310] Created kms service
I20260620 04:26:34.447584 392 mini_ranger_kms.cc:326] Created kudu user
I20260620 04:26:34.488786 392 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 04:26:34 dist-test-slave-nk6d krb5kdc[11149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929592, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 04:26:34.865873 392 mini_ranger_kms.cc:384] Added ranger policy
I20260620 04:26:34.866020 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taske80qNG/build/debug/bin/postgresql.jar:/tmp/dist-test-taske80qNG/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taske80qNG/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 04:26:34.866039 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260620 04:26:34.869630 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11542
I20260620 04:26:34.869767 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:59845
Jun 20, 2026 4:26:35 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 4:26:35 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 4:26:36 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 4:26:36 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 4:26:36 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 4:26:36 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260620 04:26:36.921788 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.774s user 0.011s sys 0.136s
I20260620 04:26:36.921984 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 04:26:36.922116 392 mini_ranger_kms.cc:401] 127.0.98.20:59845/kms/v1/keys
Jun 20 04:26:37 dist-test-slave-nk6d krb5kdc[11149](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781929597, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 04:26:37 dist-test-slave-nk6d krb5kdc[11149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929597, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 20 04:26:37 dist-test-slave-nk6d krb5kdc[11149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929597, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 20 04:26:40 dist-test-slave-nk6d krb5kdc[11149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929600, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 04:26:40.105151 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38773
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38773
--ranger_config_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:59845/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:26:40.213800 11593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:26:40.214058 11593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:26:40.214097 11593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:26:40.218465 11593 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:26:40.218528 11593 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:26:40.218544 11593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:26:40.218559 11593 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:26:40.218573 11593 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:26:40.223912 11593 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:59845/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38773
--ranger_config_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38773
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11593
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:26:40.224959 11593 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:26:40.225966 11593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:26:40.234304 11601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:26:40.234395 11599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:26:40.234304 11598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:26:40.234699 11593 server_base.cc:1034] running on GCE node
I20260620 04:26:40.235313 11593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:26:40.236480 11593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:26:40.237637 11593 hybrid_clock.cc:648] HybridClock initialized: now 1781929600237616 us; error 40 us; skew 500 ppm
Jun 20 04:26:40 dist-test-slave-nk6d krb5kdc[11149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929600, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:26:40.245110 11593 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 04:26:40.246029 11593 webserver.cc:460] Webserver started at http://127.0.98.62:40189/ using document root <none> and password file <none>
I20260620 04:26:40.246538 11593 fs_manager.cc:359] Metadata directory not provided
I20260620 04:26:40.246592 11593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:26:40.246774 11593 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 04:26:40 dist-test-slave-nk6d krb5kdc[11149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929600, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 04:26:40.305509 11593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/minicluster-data/master-0/data/instance:
uuid: "bd9d714c690246bcb7d076d0547c34cb"
format_stamp: "Formatted at 2026-06-20 04:26:40 on dist-test-slave-nk6d"
server_key: "19d43d57d73e8b826410979885084237"
server_key_iv: "1219a838da9c56a7211708bef771934d"
server_key_version: "kuduclusterkey@0"
I20260620 04:26:40.306078 11593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/minicluster-data/master-0/wal/instance:
uuid: "bd9d714c690246bcb7d076d0547c34cb"
format_stamp: "Formatted at 2026-06-20 04:26:40 on dist-test-slave-nk6d"
server_key: "19d43d57d73e8b826410979885084237"
server_key_iv: "1219a838da9c56a7211708bef771934d"
server_key_version: "kuduclusterkey@0"
I20260620 04:26:40.309821 11593 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 04:26:40.323400 11608 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:26:40.324770 11593 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:26:40.324911 11593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/minicluster-data/master-0/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/minicluster-data/master-0/wal
uuid: "bd9d714c690246bcb7d076d0547c34cb"
format_stamp: "Formatted at 2026-06-20 04:26:40 on dist-test-slave-nk6d"
server_key: "19d43d57d73e8b826410979885084237"
server_key_iv: "1219a838da9c56a7211708bef771934d"
server_key_version: "kuduclusterkey@0"
I20260620 04:26:40.325040 11593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:26:40.340590 11593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:26:40.343276 11593 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:26:40.343443 11593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:26:40.351097 11593 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38773
I20260620 04:26:40.351126 11660 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38773 every 8 connection(s)
I20260620 04:26:40.352123 11593 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:26:40.354772 11661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:26:40.360203 11661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd9d714c690246bcb7d076d0547c34cb: Bootstrap starting.
I20260620 04:26:40.361735 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 11593
I20260620 04:26:40.361850 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/minicluster-data/master-0/wal/instance
I20260620 04:26:40.362134 392 external_mini_cluster.cc:1428] Setting key 33fe177dfd14a1a84e3abdb2af22681d
I20260620 04:26:40.362924 11661 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd9d714c690246bcb7d076d0547c34cb: Neither blocks nor log segments found. Creating new log.
I20260620 04:26:40.363786 11661 log.cc:824] T 00000000000000000000000000000000 P bd9d714c690246bcb7d076d0547c34cb: Log is configured to *not* fsync() on all Append() calls
I20260620 04:26:40.365577 11661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd9d714c690246bcb7d076d0547c34cb: No bootstrap required, opened a new log
I20260620 04:26:40.369364 11661 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd9d714c690246bcb7d076d0547c34cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd9d714c690246bcb7d076d0547c34cb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38773 } }
I20260620 04:26:40.369570 11661 raft_consensus.cc:374] T 00000000000000000000000000000000 P bd9d714c690246bcb7d076d0547c34cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:26:40.369635 11661 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd9d714c690246bcb7d076d0547c34cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd9d714c690246bcb7d076d0547c34cb, State: Initialized, Role: FOLLOWER
I20260620 04:26:40.370064 11661 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd9d714c690246bcb7d076d0547c34cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd9d714c690246bcb7d076d0547c34cb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38773 } }
I20260620 04:26:40.370199 11661 raft_consensus.cc:388] T 00000000000000000000000000000000 P bd9d714c690246bcb7d076d0547c34cb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:26:40.370273 11661 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd9d714c690246bcb7d076d0547c34cb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:26:40.370368 11661 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd9d714c690246bcb7d076d0547c34cb [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:26:40.371241 11661 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd9d714c690246bcb7d076d0547c34cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd9d714c690246bcb7d076d0547c34cb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38773 } }
I20260620 04:26:40.371554 11661 leader_election.cc:302] T 00000000000000000000000000000000 P bd9d714c690246bcb7d076d0547c34cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd9d714c690246bcb7d076d0547c34cb; no voters:
I20260620 04:26:40.371857 11661 leader_election.cc:288] T 00000000000000000000000000000000 P bd9d714c690246bcb7d076d0547c34cb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:26:40.372025 11666 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd9d714c690246bcb7d076d0547c34cb [term 1 FOLLOWER]: Leader election won for term 1
Jun 20 04:26:40 dist-test-slave-nk6d krb5kdc[11149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929600, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:26:40.372253 11666 raft_consensus.cc:686] T 00000000000000000000000000000000 P bd9d714c690246bcb7d076d0547c34cb [term 1 LEADER]: Becoming Leader. State: Replica: bd9d714c690246bcb7d076d0547c34cb, State: Running, Role: LEADER
I20260620 04:26:40.372545 11666 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd9d714c690246bcb7d076d0547c34cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd9d714c690246bcb7d076d0547c34cb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38773 } }
I20260620 04:26:40.373196 11661 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd9d714c690246bcb7d076d0547c34cb [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:26:40.374238 11667 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd9d714c690246bcb7d076d0547c34cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bd9d714c690246bcb7d076d0547c34cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd9d714c690246bcb7d076d0547c34cb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38773 } } }
I20260620 04:26:40.374346 11667 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd9d714c690246bcb7d076d0547c34cb [sys.catalog]: This master's current role is: LEADER
I20260620 04:26:40.374531 11667 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd9d714c690246bcb7d076d0547c34cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd9d714c690246bcb7d076d0547c34cb. Latest consensus state: current_term: 1 leader_uuid: "bd9d714c690246bcb7d076d0547c34cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd9d714c690246bcb7d076d0547c34cb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38773 } } }
I20260620 04:26:40.374588 11667 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd9d714c690246bcb7d076d0547c34cb [sys.catalog]: This master's current role is: LEADER
I20260620 04:26:40.374933 11675 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:26:40.375437 11661 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 04:26:40.378732 11664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:26:40.363294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54004 (local address 127.0.98.62:38773)
0620 04:26:40.363761 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0620 04:26:40.363780 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 04:26:40.363822 (+ 42us) server_negotiation.cc:407] Connection header received
0620 04:26:40.364684 (+ 862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:26:40.364709 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:26:40.365123 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:26:40.365481 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:26:40.366295 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:26:40.367797 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:26:40.369040 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:26:40.369789 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:26:40.370567 (+ 778us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:26:40.372913 (+ 2346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:26:40.372920 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:26:40.372935 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:26:40.372965 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:26:40.374906 (+ 1941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:26:40.375620 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:26:40.375626 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:26:40.375632 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:26:40.375757 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:26:40.376078 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:26:40.376082 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:26:40.376084 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:26:40.376572 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:26:40.376761 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0620 04:26:40.377548 (+ 787us) server_negotiation.cc:299] Negotiation successful
0620 04:26:40.377816 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":156,"threads_started":1}
I20260620 04:26:40.378756 11675 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:26:40.393409 11675 catalog_manager.cc:1347] Generated new cluster ID: f167158e91af418998c1ddc60c8c0e1b
I20260620 04:26:40.393474 11675 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:26:40.437917 11675 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:26:40.439249 11675 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:26:40.445204 11675 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bd9d714c690246bcb7d076d0547c34cb: Generated new TSK 0
I20260620 04:26:40.446062 11675 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 04:26:41.173286 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:59845/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38773
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:26:41.316566 11706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:26:41.316819 11706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:26:41.316882 11706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:26:41.320971 11706 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:26:41.321027 11706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:26:41.321105 11706 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 04:26:41.327626 11706 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:59845/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38773
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11706
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:26:41.328944 11706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:26:41.330262 11706 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:26:41.341049 11716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:26:41.341027 11714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:26:41.341259 11713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:26:41.341684 11706 server_base.cc:1034] running on GCE node
I20260620 04:26:41.342065 11706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:26:41.342701 11706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:26:41.343864 11706 hybrid_clock.cc:648] HybridClock initialized: now 1781929601343842 us; error 40 us; skew 500 ppm
Jun 20 04:26:41 dist-test-slave-nk6d krb5kdc[11149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929601, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:26:41.346529 11706 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 04:26:41.347450 11706 webserver.cc:460] Webserver started at http://127.0.98.1:35899/ using document root <none> and password file <none>
I20260620 04:26:41.348003 11706 fs_manager.cc:359] Metadata directory not provided
I20260620 04:26:41.348078 11706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:26:41.348300 11706 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 04:26:41 dist-test-slave-nk6d krb5kdc[11149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929601, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 04:26:41.365046 11706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/minicluster-data/ts-0/data/instance:
uuid: "cf16724c5e234b0ea15dcefd619c912d"
format_stamp: "Formatted at 2026-06-20 04:26:41 on dist-test-slave-nk6d"
server_key: "7dae36af721f1ccde3301d7efdeabb99"
server_key_iv: "9a6529edfd16354d7ceb7ecfc1ecc417"
server_key_version: "kuduclusterkey@0"
I20260620 04:26:41.365844 11706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/minicluster-data/ts-0/wal/instance:
uuid: "cf16724c5e234b0ea15dcefd619c912d"
format_stamp: "Formatted at 2026-06-20 04:26:41 on dist-test-slave-nk6d"
server_key: "7dae36af721f1ccde3301d7efdeabb99"
server_key_iv: "9a6529edfd16354d7ceb7ecfc1ecc417"
server_key_version: "kuduclusterkey@0"
I20260620 04:26:41.373363 11706 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260620 04:26:41.393441 11723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:26:41.395087 11706 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260620 04:26:41.395207 11706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/minicluster-data/ts-0/wal
uuid: "cf16724c5e234b0ea15dcefd619c912d"
format_stamp: "Formatted at 2026-06-20 04:26:41 on dist-test-slave-nk6d"
server_key: "7dae36af721f1ccde3301d7efdeabb99"
server_key_iv: "9a6529edfd16354d7ceb7ecfc1ecc417"
server_key_version: "kuduclusterkey@0"
I20260620 04:26:41.395331 11706 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:26:41.430218 11706 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:26:41.433847 11706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:26:41.434079 11706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:26:41.434782 11706 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:26:41.435844 11706 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:26:41.435931 11706 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:26:41.436033 11706 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:26:41.436080 11706 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:26:41.446668 11706 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45265
I20260620 04:26:41.447842 11706 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/minicluster-data/ts-0/data/info.pb
I20260620 04:26:41.450101 11836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45265 every 8 connection(s)
I20260620 04:26:41.452613 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 11706
I20260620 04:26:41.452732 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/minicluster-data/ts-0/wal/instance
I20260620 04:26:41.453058 392 external_mini_cluster.cc:1428] Setting key 57841c85583536e7c91a3754d7c091b3
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Jun 20 04:26:41 dist-test-slave-nk6d krb5kdc[11149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929601, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:26:41.482894 11842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:26:41.455052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34869 (local address 127.0.98.62:38773)
0620 04:26:41.455314 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0620 04:26:41.455319 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:26:41.463830 (+ 8511us) server_negotiation.cc:407] Connection header received
0620 04:26:41.467245 (+ 3415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:26:41.467249 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:26:41.467310 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:26:41.467368 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:26:41.469126 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:26:41.469980 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:26:41.473672 (+ 3692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:26:41.474025 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:26:41.474097 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:26:41.477152 (+ 3055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:26:41.477156 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:26:41.477158 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:26:41.477188 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:26:41.478914 (+ 1726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:26:41.481019 (+ 2105us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:26:41.481023 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:26:41.481025 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:26:41.481125 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:26:41.481523 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:26:41.481544 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:26:41.481546 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:26:41.481778 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:26:41.481855 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 04:26:41.482568 (+ 713us) server_negotiation.cc:299] Negotiation successful
0620 04:26:41.482711 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":90,"threads_started":1}
I20260620 04:26:41.484804 11841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:26:41.453991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38773 (local address 127.0.98.1:34869)
0620 04:26:41.463631 (+ 9640us) negotiation.cc:107] Waiting for socket to connect
0620 04:26:41.463705 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0620 04:26:41.466966 (+ 3261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:26:41.467821 (+ 855us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:26:41.467831 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:26:41.468181 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:26:41.468954 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:26:41.468974 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:26:41.470138 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:26:41.470141 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:26:41.473497 (+ 3356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:26:41.473508 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:26:41.474309 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:26:41.474313 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:26:41.474583 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:26:41.475238 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:26:41.475258 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:26:41.476944 (+ 1686us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:26:41.479048 (+ 2104us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:26:41.479053 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:26:41.479070 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:26:41.479387 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:26:41.481231 (+ 1844us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:26:41.481233 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:26:41.481235 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:26:41.481421 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:26:41.481944 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:26:41.481952 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:26:41.482297 (+ 345us) client_negotiation.cc:769] Sending connection context
0620 04:26:41.483328 (+ 1031us) client_negotiation.cc:240] Negotiation successful
0620 04:26:41.483680 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9343,"thread_start_us":130,"threads_started":1}
I20260620 04:26:41.489483 11837 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38773
I20260620 04:26:41.489750 11837 heartbeater.cc:464] Registering TS with master...
I20260620 04:26:41.490331 11837 heartbeater.cc:511] Master 127.0.98.62:38773 requested a full tablet report, sending...
I20260620 04:26:41.492100 11624 ts_manager.cc:194] Registered new tserver with Master: cf16724c5e234b0ea15dcefd619c912d (127.0.98.1:45265)
I20260620 04:26:41.493582 11624 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:34869
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 04:26:41.504753 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:59845/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38773
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 04:26:41 dist-test-slave-nk6d krb5kdc[11149](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781929601, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 04:26:41.670436 11845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:26:41.670876 11845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:26:41.670974 11845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:26:41.677548 11845 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:26:41.677665 11845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:26:41.677789 11845 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 04:26:41.685719 11845 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:59845/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38773
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11845
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:26:41.687179 11845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:26:41.688563 11845 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:26:41.701046 11851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:26:41.701505 11852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:26:41.702008 11854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:26:41.702765 11845 server_base.cc:1034] running on GCE node
I20260620 04:26:41.703212 11845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:26:41.703912 11845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:26:41.705106 11845 hybrid_clock.cc:648] HybridClock initialized: now 1781929601705076 us; error 45 us; skew 500 ppm
Jun 20 04:26:41 dist-test-slave-nk6d krb5kdc[11149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929601, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:26:41.708632 11845 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 04:26:41.709709 11845 webserver.cc:460] Webserver started at http://127.0.98.2:37853/ using document root <none> and password file <none>
I20260620 04:26:41.710424 11845 fs_manager.cc:359] Metadata directory not provided
I20260620 04:26:41.710556 11845 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:26:41.710824 11845 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 04:26:41 dist-test-slave-nk6d krb5kdc[11149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929601, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 04:26:41.730974 11845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/minicluster-data/ts-1/data/instance:
uuid: "bfa7fe045ef14276ae5499bd373f13bb"
format_stamp: "Formatted at 2026-06-20 04:26:41 on dist-test-slave-nk6d"
server_key: "7d5db60a5b24e6b0012c0ff0989abf20"
server_key_iv: "cc77dcbe8c31a469e695c8547038cce9"
server_key_version: "kuduclusterkey@0"
I20260620 04:26:41.731747 11845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/minicluster-data/ts-1/wal/instance:
uuid: "bfa7fe045ef14276ae5499bd373f13bb"
format_stamp: "Formatted at 2026-06-20 04:26:41 on dist-test-slave-nk6d"
server_key: "7d5db60a5b24e6b0012c0ff0989abf20"
server_key_iv: "cc77dcbe8c31a469e695c8547038cce9"
server_key_version: "kuduclusterkey@0"
I20260620 04:26:41.742573 11845 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.002s sys 0.003s
I20260620 04:26:41.765239 11861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:26:41.766875 11845 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 04:26:41.766999 11845 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/minicluster-data/ts-1/wal
uuid: "bfa7fe045ef14276ae5499bd373f13bb"
format_stamp: "Formatted at 2026-06-20 04:26:41 on dist-test-slave-nk6d"
server_key: "7d5db60a5b24e6b0012c0ff0989abf20"
server_key_iv: "cc77dcbe8c31a469e695c8547038cce9"
server_key_version: "kuduclusterkey@0"
I20260620 04:26:41.767129 11845 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:26:41.790202 11845 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:26:41.793708 11845 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:26:41.793936 11845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:26:41.794688 11845 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:26:41.795785 11845 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:26:41.795883 11845 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:26:41.796000 11845 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:26:41.796069 11845 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:26:41.809324 11845 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34851
I20260620 04:26:41.809366 11974 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34851 every 8 connection(s)
I20260620 04:26:41.810503 11845 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/minicluster-data/ts-1/data/info.pb
I20260620 04:26:41.815001 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 11845
I20260620 04:26:41.815135 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/minicluster-data/ts-1/wal/instance
I20260620 04:26:41.815421 392 external_mini_cluster.cc:1428] Setting key 57779c20710ecc9a2b0625dab2b0950a
Jun 20 04:26:41 dist-test-slave-nk6d krb5kdc[11149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929601, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:26:41.835124 11842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:26:41.812976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60977 (local address 127.0.98.62:38773)
0620 04:26:41.813119 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 04:26:41.813123 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:26:41.819232 (+ 6109us) server_negotiation.cc:407] Connection header received
0620 04:26:41.820643 (+ 1411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:26:41.820646 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:26:41.820698 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:26:41.820797 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:26:41.822350 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:26:41.823198 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:26:41.826159 (+ 2961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:26:41.826536 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:26:41.826639 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:26:41.831015 (+ 4376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:26:41.831020 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:26:41.831023 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:26:41.831055 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:26:41.832823 (+ 1768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:26:41.833410 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:26:41.833414 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:26:41.833416 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:26:41.833505 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:26:41.833868 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:26:41.833872 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:26:41.833874 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:26:41.834087 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:26:41.834194 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 04:26:41.834809 (+ 615us) server_negotiation.cc:299] Negotiation successful
0620 04:26:41.834943 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 04:26:41.836277 11977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:26:41.813484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38773 (local address 127.0.98.2:60977)
0620 04:26:41.819064 (+ 5580us) negotiation.cc:107] Waiting for socket to connect
0620 04:26:41.819138 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0620 04:26:41.820297 (+ 1159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:26:41.820990 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:26:41.821000 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:26:41.821367 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:26:41.822023 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:26:41.822041 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:26:41.824592 (+ 2551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:26:41.824596 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:26:41.826025 (+ 1429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:26:41.826037 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:26:41.828065 (+ 2028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:26:41.828069 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:26:41.828338 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:26:41.829081 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:26:41.829111 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:26:41.830835 (+ 1724us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:26:41.832934 (+ 2099us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:26:41.832941 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:26:41.832958 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:26:41.833304 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:26:41.833593 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:26:41.833596 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:26:41.833598 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:26:41.833775 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:26:41.834208 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:26:41.834215 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:26:41.834546 (+ 331us) client_negotiation.cc:769] Sending connection context
0620 04:26:41.834744 (+ 198us) client_negotiation.cc:240] Negotiation successful
0620 04:26:41.835087 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5333,"thread_start_us":124,"threads_started":1}
I20260620 04:26:41.837806 11975 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38773
I20260620 04:26:41.838044 11975 heartbeater.cc:464] Registering TS with master...
I20260620 04:26:41.838649 11975 heartbeater.cc:511] Master 127.0.98.62:38773 requested a full tablet report, sending...
I20260620 04:26:41.839830 11624 ts_manager.cc:194] Registered new tserver with Master: bfa7fe045ef14276ae5499bd373f13bb (127.0.98.2:34851)
I20260620 04:26:41.840668 11624 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:60977
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 04:26:41.867256 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:59845/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38773
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:26:42.029189 11980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:26:42.029696 11980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:26:42.029827 11980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:26:42.035948 11980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:26:42.036111 11980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:26:42.036254 11980 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 04:26:42.043859 11980 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:59845/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38773
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11980
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:26:42.045485 11980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:26:42.046860 11980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
Jun 20 04:26:42 dist-test-slave-nk6d krb5kdc[11149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929601, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260620 04:26:42.084467 11986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:26:42.088213 11988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:26:42.088708 11985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:26:42.092805 11980 server_base.cc:1034] running on GCE node
I20260620 04:26:42.093348 11980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:26:42.094229 11980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:26:42.109289 11980 hybrid_clock.cc:648] HybridClock initialized: now 1781929602109078 us; error 260 us; skew 500 ppm
Jun 20 04:26:42 dist-test-slave-nk6d krb5kdc[11149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929602, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:26:42.115234 11980 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 04:26:42.116592 11980 webserver.cc:460] Webserver started at http://127.0.98.3:34645/ using document root <none> and password file <none>
I20260620 04:26:42.117667 11980 fs_manager.cc:359] Metadata directory not provided
I20260620 04:26:42.117813 11980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:26:42.118294 11980 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 04:26:42 dist-test-slave-nk6d krb5kdc[11149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929602, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 04:26:42.155329 11980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/minicluster-data/ts-2/data/instance:
uuid: "22aa4c1e667043fb877f2ac1acebdc02"
format_stamp: "Formatted at 2026-06-20 04:26:42 on dist-test-slave-nk6d"
server_key: "d210436f6500d785a88011cc12576bba"
server_key_iv: "93efe0423c7afb3a9ace252343866545"
server_key_version: "kuduclusterkey@0"
I20260620 04:26:42.157861 11980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/minicluster-data/ts-2/wal/instance:
uuid: "22aa4c1e667043fb877f2ac1acebdc02"
format_stamp: "Formatted at 2026-06-20 04:26:42 on dist-test-slave-nk6d"
server_key: "d210436f6500d785a88011cc12576bba"
server_key_iv: "93efe0423c7afb3a9ace252343866545"
server_key_version: "kuduclusterkey@0"
I20260620 04:26:42.163590 11980 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 04:26:42.183708 11995 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:26:42.185546 11980 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.004s sys 0.000s
I20260620 04:26:42.185770 11980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/minicluster-data/ts-2/wal
uuid: "22aa4c1e667043fb877f2ac1acebdc02"
format_stamp: "Formatted at 2026-06-20 04:26:42 on dist-test-slave-nk6d"
server_key: "d210436f6500d785a88011cc12576bba"
server_key_iv: "93efe0423c7afb3a9ace252343866545"
server_key_version: "kuduclusterkey@0"
I20260620 04:26:42.185982 11980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:26:42.217247 11980 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:26:42.221186 11980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:26:42.221390 11980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:26:42.222070 11980 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:26:42.223124 11980 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:26:42.223193 11980 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:26:42.223333 11980 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:26:42.223377 11980 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:26:42.237275 11980 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40563
I20260620 04:26:42.238530 11980 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/minicluster-data/ts-2/data/info.pb
I20260620 04:26:42.239362 12108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40563 every 8 connection(s)
I20260620 04:26:42.243101 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 11980
I20260620 04:26:42.243225 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/minicluster-data/ts-2/wal/instance
I20260620 04:26:42.243531 392 external_mini_cluster.cc:1428] Setting key f83a69454f2afdaf82aa3be6387d4190
Jun 20 04:26:42 dist-test-slave-nk6d krb5kdc[11149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929602, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:26:42.256914 11842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:26:42.242828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38863 (local address 127.0.98.62:38773)
0620 04:26:42.242958 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 04:26:42.242962 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:26:42.244176 (+ 1214us) server_negotiation.cc:407] Connection header received
0620 04:26:42.245325 (+ 1149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:26:42.245328 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:26:42.245379 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:26:42.245426 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:26:42.247044 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:26:42.247837 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:26:42.249568 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:26:42.249929 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:26:42.250007 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:26:42.252919 (+ 2912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:26:42.252923 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:26:42.252926 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:26:42.252953 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:26:42.254515 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:26:42.255118 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:26:42.255121 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:26:42.255123 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:26:42.255207 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:26:42.255594 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:26:42.255598 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:26:42.255599 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:26:42.255845 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:26:42.255935 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 04:26:42.256622 (+ 687us) server_negotiation.cc:299] Negotiation successful
0620 04:26:42.256761 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 04:26:42.258709 12111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:26:42.241774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38773 (local address 127.0.98.3:38863)
0620 04:26:42.244033 (+ 2259us) negotiation.cc:107] Waiting for socket to connect
0620 04:26:42.244087 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 04:26:42.245101 (+ 1014us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:26:42.245843 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:26:42.245854 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:26:42.246189 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:26:42.246879 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:26:42.246900 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:26:42.248010 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:26:42.248013 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:26:42.249431 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:26:42.249442 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:26:42.250202 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:26:42.250207 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:26:42.250401 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:26:42.251045 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:26:42.251066 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:26:42.252742 (+ 1676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:26:42.254653 (+ 1911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:26:42.254660 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:26:42.254676 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:26:42.255007 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:26:42.255306 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:26:42.255309 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:26:42.255311 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:26:42.255482 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:26:42.256026 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:26:42.256032 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:26:42.256363 (+ 331us) client_negotiation.cc:769] Sending connection context
0620 04:26:42.257314 (+ 951us) client_negotiation.cc:240] Negotiation successful
0620 04:26:42.257651 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2017,"thread_start_us":132,"threads_started":1}
I20260620 04:26:42.260172 12109 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38773
I20260620 04:26:42.260411 12109 heartbeater.cc:464] Registering TS with master...
I20260620 04:26:42.260969 12109 heartbeater.cc:511] Master 127.0.98.62:38773 requested a full tablet report, sending...
I20260620 04:26:42.262231 11624 ts_manager.cc:194] Registered new tserver with Master: 22aa4c1e667043fb877f2ac1acebdc02 (127.0.98.3:40563)
I20260620 04:26:42.262972 11624 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:38863
I20260620 04:26:42.269219 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 04:26:42.286425 11842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:26:42.276451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54014 (local address 127.0.98.62:38773)
0620 04:26:42.276571 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 04:26:42.276577 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 04:26:42.277262 (+ 685us) server_negotiation.cc:407] Connection header received
0620 04:26:42.277458 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:26:42.277462 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:26:42.277515 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:26:42.277570 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:26:42.278571 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:26:42.279366 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:26:42.280627 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:26:42.280962 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:26:42.281043 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:26:42.282824 (+ 1781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:26:42.282828 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:26:42.282831 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:26:42.282857 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:26:42.284571 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:26:42.285069 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:26:42.285073 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:26:42.285075 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:26:42.285158 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:26:42.285463 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:26:42.285467 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:26:42.285469 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:26:42.285692 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:26:42.285802 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 04:26:42.286144 (+ 342us) server_negotiation.cc:299] Negotiation successful
0620 04:26:42.286257 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 04:26:42.290381 11624 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54014:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 04:26:42.294998 11624 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 04:26:42.324538 12124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:26:42.310200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34851 (local address 127.0.0.1:43236)
0620 04:26:42.314228 (+ 4028us) negotiation.cc:107] Waiting for socket to connect
0620 04:26:42.314240 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 04:26:42.314322 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:26:42.315246 (+ 924us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:26:42.315248 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:26:42.315262 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:26:42.315346 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:26:42.315350 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:26:42.321260 (+ 5910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:26:42.321263 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:26:42.322566 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:26:42.322577 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:26:42.323933 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:26:42.323937 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:26:42.324084 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:26:42.324131 (+ 47us) client_negotiation.cc:769] Sending connection context
0620 04:26:42.324246 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 04:26:42.324383 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3972,"thread_start_us":74,"threads_started":1}
I20260620 04:26:42.327176 12126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:26:42.307051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40563 (local address 127.0.0.1:52046)
0620 04:26:42.311839 (+ 4788us) negotiation.cc:107] Waiting for socket to connect
0620 04:26:42.311892 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 04:26:42.312041 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:26:42.314991 (+ 2950us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:26:42.314997 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:26:42.315018 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:26:42.315145 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:26:42.315152 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:26:42.318628 (+ 3476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:26:42.318631 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:26:42.320107 (+ 1476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:26:42.320118 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:26:42.326795 (+ 6677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:26:42.326798 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:26:42.326889 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:26:42.326909 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 04:26:42.326970 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 04:26:42.327034 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4603,"thread_start_us":93,"threads_started":1}
I20260620 04:26:42.328617 12127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:26:42.310554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43236 (local address 127.0.98.2:34851)
0620 04:26:42.312393 (+ 1839us) server_negotiation.cc:206] Beginning negotiation
0620 04:26:42.312409 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 04:26:42.314411 (+ 2002us) server_negotiation.cc:407] Connection header received
0620 04:26:42.314492 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:26:42.314497 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:26:42.314696 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:26:42.314831 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:26:42.315450 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:26:42.321130 (+ 5680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:26:42.322722 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:26:42.323576 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:26:42.323691 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:26:42.323866 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0620 04:26:42.328350 (+ 4484us) server_negotiation.cc:299] Negotiation successful
0620 04:26:42.328477 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1702,"thread_start_us":57,"threads_started":1}
I20260620 04:26:42.330209 12123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:26:42.308116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52046 (local address 127.0.98.3:40563)
0620 04:26:42.312865 (+ 4749us) server_negotiation.cc:206] Beginning negotiation
0620 04:26:42.312875 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 04:26:42.312898 (+ 23us) server_negotiation.cc:407] Connection header received
0620 04:26:42.312972 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:26:42.312977 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:26:42.313193 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:26:42.313333 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:26:42.317373 (+ 4040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:26:42.318493 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:26:42.324920 (+ 6427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:26:42.326495 (+ 1575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:26:42.326587 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:26:42.326736 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 04:26:42.328933 (+ 2197us) server_negotiation.cc:299] Negotiation successful
0620 04:26:42.329079 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4649,"thread_start_us":96,"threads_started":1}
I20260620 04:26:42.332535 11909 tablet_service.cc:1467] Processing CreateTablet for tablet 90e159376c9b41fd8d10a5efd249e817 (DEFAULT_TABLE table=test-table [id=27088cf1ab614d9087ae25a8dda470e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:26:42.333315 12043 tablet_service.cc:1467] Processing CreateTablet for tablet 90e159376c9b41fd8d10a5efd249e817 (DEFAULT_TABLE table=test-table [id=27088cf1ab614d9087ae25a8dda470e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:26:42.333734 11909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90e159376c9b41fd8d10a5efd249e817. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:26:42.334326 12043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90e159376c9b41fd8d10a5efd249e817. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:26:42.342275 12122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:26:42.309574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45265 (local address 127.0.0.1:50450)
0620 04:26:42.312534 (+ 2960us) negotiation.cc:107] Waiting for socket to connect
0620 04:26:42.312547 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:26:42.312624 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:26:42.326204 (+ 13580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:26:42.326207 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:26:42.326224 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:26:42.326313 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:26:42.326319 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:26:42.327638 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:26:42.327641 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:26:42.328799 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:26:42.328810 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:26:42.330038 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:26:42.330042 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:26:42.341920 (+ 11878us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:26:42.341952 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 04:26:42.342031 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 04:26:42.342128 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2902,"thread_start_us":104,"threads_started":1}
I20260620 04:26:42.343037 12125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:26:42.309718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50450 (local address 127.0.98.1:45265)
0620 04:26:42.325540 (+ 15822us) server_negotiation.cc:206] Beginning negotiation
0620 04:26:42.325547 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 04:26:42.325573 (+ 26us) server_negotiation.cc:407] Connection header received
0620 04:26:42.325657 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:26:42.325663 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:26:42.325902 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:26:42.326055 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:26:42.326439 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:26:42.327508 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:26:42.328966 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:26:42.329759 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:26:42.329854 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:26:42.329976 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 04:26:42.342769 (+ 12793us) server_negotiation.cc:299] Negotiation successful
0620 04:26:42.342898 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15701,"thread_start_us":100,"threads_started":1}
I20260620 04:26:42.344381 12129 tablet_bootstrap.cc:492] T 90e159376c9b41fd8d10a5efd249e817 P 22aa4c1e667043fb877f2ac1acebdc02: Bootstrap starting.
I20260620 04:26:42.344381 12128 tablet_bootstrap.cc:492] T 90e159376c9b41fd8d10a5efd249e817 P bfa7fe045ef14276ae5499bd373f13bb: Bootstrap starting.
I20260620 04:26:42.346135 11760 tablet_service.cc:1467] Processing CreateTablet for tablet 90e159376c9b41fd8d10a5efd249e817 (DEFAULT_TABLE table=test-table [id=27088cf1ab614d9087ae25a8dda470e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:26:42.346721 12128 tablet_bootstrap.cc:654] T 90e159376c9b41fd8d10a5efd249e817 P bfa7fe045ef14276ae5499bd373f13bb: Neither blocks nor log segments found. Creating new log.
I20260620 04:26:42.347199 12129 tablet_bootstrap.cc:654] T 90e159376c9b41fd8d10a5efd249e817 P 22aa4c1e667043fb877f2ac1acebdc02: Neither blocks nor log segments found. Creating new log.
I20260620 04:26:42.347270 11760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90e159376c9b41fd8d10a5efd249e817. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:26:42.348519 12129 log.cc:824] T 90e159376c9b41fd8d10a5efd249e817 P 22aa4c1e667043fb877f2ac1acebdc02: Log is configured to *not* fsync() on all Append() calls
I20260620 04:26:42.350696 12129 tablet_bootstrap.cc:492] T 90e159376c9b41fd8d10a5efd249e817 P 22aa4c1e667043fb877f2ac1acebdc02: No bootstrap required, opened a new log
I20260620 04:26:42.350888 12129 ts_tablet_manager.cc:1397] T 90e159376c9b41fd8d10a5efd249e817 P 22aa4c1e667043fb877f2ac1acebdc02: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260620 04:26:42.352486 12128 log.cc:824] T 90e159376c9b41fd8d10a5efd249e817 P bfa7fe045ef14276ae5499bd373f13bb: Log is configured to *not* fsync() on all Append() calls
I20260620 04:26:42.354756 12129 raft_consensus.cc:348] T 90e159376c9b41fd8d10a5efd249e817 P 22aa4c1e667043fb877f2ac1acebdc02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22aa4c1e667043fb877f2ac1acebdc02" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40563 } } peers { permanent_uuid: "cf16724c5e234b0ea15dcefd619c912d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45265 } } peers { permanent_uuid: "bfa7fe045ef14276ae5499bd373f13bb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34851 } }
I20260620 04:26:42.355016 12129 raft_consensus.cc:374] T 90e159376c9b41fd8d10a5efd249e817 P 22aa4c1e667043fb877f2ac1acebdc02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:26:42.355069 12129 raft_consensus.cc:729] T 90e159376c9b41fd8d10a5efd249e817 P 22aa4c1e667043fb877f2ac1acebdc02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22aa4c1e667043fb877f2ac1acebdc02, State: Initialized, Role: FOLLOWER
I20260620 04:26:42.356957 12129 consensus_queue.cc:260] T 90e159376c9b41fd8d10a5efd249e817 P 22aa4c1e667043fb877f2ac1acebdc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22aa4c1e667043fb877f2ac1acebdc02" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40563 } } peers { permanent_uuid: "cf16724c5e234b0ea15dcefd619c912d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45265 } } peers { permanent_uuid: "bfa7fe045ef14276ae5499bd373f13bb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34851 } }
I20260620 04:26:42.358228 12129 ts_tablet_manager.cc:1428] T 90e159376c9b41fd8d10a5efd249e817 P 22aa4c1e667043fb877f2ac1acebdc02: Time spent starting tablet: real 0.007s user 0.000s sys 0.005s
I20260620 04:26:42.358624 12109 heartbeater.cc:503] Master 127.0.98.62:38773 was elected leader, sending a full tablet report...
I20260620 04:26:42.367069 12132 raft_consensus.cc:482] T 90e159376c9b41fd8d10a5efd249e817 P 22aa4c1e667043fb877f2ac1acebdc02 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:26:42.367326 12132 raft_consensus.cc:504] T 90e159376c9b41fd8d10a5efd249e817 P 22aa4c1e667043fb877f2ac1acebdc02 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22aa4c1e667043fb877f2ac1acebdc02" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40563 } } peers { permanent_uuid: "cf16724c5e234b0ea15dcefd619c912d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45265 } } peers { permanent_uuid: "bfa7fe045ef14276ae5499bd373f13bb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34851 } }
I20260620 04:26:42.370162 12132 leader_election.cc:288] T 90e159376c9b41fd8d10a5efd249e817 P 22aa4c1e667043fb877f2ac1acebdc02 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cf16724c5e234b0ea15dcefd619c912d (127.0.98.1:45265), bfa7fe045ef14276ae5499bd373f13bb (127.0.98.2:34851)
I20260620 04:26:42.371373 12133 tablet_bootstrap.cc:492] T 90e159376c9b41fd8d10a5efd249e817 P cf16724c5e234b0ea15dcefd619c912d: Bootstrap starting.
I20260620 04:26:42.373173 12111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:26:42.368742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45265 (local address 127.0.98.3:56415)
0620 04:26:42.368968 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0620 04:26:42.368983 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 04:26:42.369062 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:26:42.369610 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:26:42.369613 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:26:42.369644 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:26:42.369747 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:26:42.369753 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:26:42.370677 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:26:42.370680 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:26:42.371786 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:26:42.371795 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:26:42.372754 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:26:42.372758 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:26:42.372855 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:26:42.372905 (+ 50us) client_negotiation.cc:769] Sending connection context
0620 04:26:42.372968 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 04:26:42.373035 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"mutex_wait_us":17}
I20260620 04:26:42.373857 12125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:26:42.369211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56415 (local address 127.0.98.1:45265)
0620 04:26:42.369329 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 04:26:42.369333 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:26:42.369346 (+ 13us) server_negotiation.cc:407] Connection header received
0620 04:26:42.369386 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:26:42.369389 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:26:42.369433 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:26:42.369502 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:26:42.369870 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:26:42.370555 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:26:42.371972 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:26:42.372538 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:26:42.372634 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:26:42.372701 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 04:26:42.373659 (+ 958us) server_negotiation.cc:299] Negotiation successful
0620 04:26:42.373754 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 04:26:42.374121 12128 tablet_bootstrap.cc:492] T 90e159376c9b41fd8d10a5efd249e817 P bfa7fe045ef14276ae5499bd373f13bb: No bootstrap required, opened a new log
I20260620 04:26:42.374423 11791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90e159376c9b41fd8d10a5efd249e817" candidate_uuid: "22aa4c1e667043fb877f2ac1acebdc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf16724c5e234b0ea15dcefd619c912d" is_pre_election: true
W20260620 04:26:42.376462 11997 leader_election.cc:341] T 90e159376c9b41fd8d10a5efd249e817 P 22aa4c1e667043fb877f2ac1acebdc02 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cf16724c5e234b0ea15dcefd619c912d (127.0.98.1:45265): Illegal state: must be running to vote when last-logged opid is not known
I20260620 04:26:42.376827 12128 ts_tablet_manager.cc:1397] T 90e159376c9b41fd8d10a5efd249e817 P bfa7fe045ef14276ae5499bd373f13bb: Time spent bootstrapping tablet: real 0.033s user 0.006s sys 0.000s
I20260620 04:26:42.379755 12127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:26:42.370006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47397 (local address 127.0.98.2:34851)
0620 04:26:42.370112 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0620 04:26:42.370135 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0620 04:26:42.370543 (+ 408us) server_negotiation.cc:407] Connection header received
0620 04:26:42.370713 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:26:42.370716 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:26:42.370758 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:26:42.370800 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:26:42.371699 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:26:42.372495 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:26:42.376996 (+ 4501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:26:42.377561 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:26:42.377636 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:26:42.377692 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 04:26:42.379559 (+ 1867us) server_negotiation.cc:299] Negotiation successful
0620 04:26:42.379640 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 04:26:42.380311 12134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:26:42.369590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34851 (local address 127.0.98.3:47397)
0620 04:26:42.370466 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0620 04:26:42.370478 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 04:26:42.370604 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:26:42.370915 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:26:42.370917 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:26:42.370932 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:26:42.371573 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:26:42.371584 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:26:42.372705 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:26:42.372708 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:26:42.376660 (+ 3952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:26:42.376671 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:26:42.377802 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:26:42.377805 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:26:42.377908 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:26:42.377930 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 04:26:42.380075 (+ 2145us) client_negotiation.cc:240] Negotiation successful
0620 04:26:42.380165 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":817,"thread_start_us":107,"threads_started":1}
I20260620 04:26:42.381117 11914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90e159376c9b41fd8d10a5efd249e817" candidate_uuid: "22aa4c1e667043fb877f2ac1acebdc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfa7fe045ef14276ae5499bd373f13bb" is_pre_election: true
I20260620 04:26:42.380749 12128 raft_consensus.cc:348] T 90e159376c9b41fd8d10a5efd249e817 P bfa7fe045ef14276ae5499bd373f13bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22aa4c1e667043fb877f2ac1acebdc02" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40563 } } peers { permanent_uuid: "cf16724c5e234b0ea15dcefd619c912d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45265 } } peers { permanent_uuid: "bfa7fe045ef14276ae5499bd373f13bb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34851 } }
I20260620 04:26:42.381510 12128 raft_consensus.cc:374] T 90e159376c9b41fd8d10a5efd249e817 P bfa7fe045ef14276ae5499bd373f13bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:26:42.381568 12128 raft_consensus.cc:729] T 90e159376c9b41fd8d10a5efd249e817 P bfa7fe045ef14276ae5499bd373f13bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfa7fe045ef14276ae5499bd373f13bb, State: Initialized, Role: FOLLOWER
I20260620 04:26:42.382012 12128 consensus_queue.cc:260] T 90e159376c9b41fd8d10a5efd249e817 P bfa7fe045ef14276ae5499bd373f13bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22aa4c1e667043fb877f2ac1acebdc02" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40563 } } peers { permanent_uuid: "cf16724c5e234b0ea15dcefd619c912d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45265 } } peers { permanent_uuid: "bfa7fe045ef14276ae5499bd373f13bb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34851 } }
I20260620 04:26:42.382084 12133 tablet_bootstrap.cc:654] T 90e159376c9b41fd8d10a5efd249e817 P cf16724c5e234b0ea15dcefd619c912d: Neither blocks nor log segments found. Creating new log.
I20260620 04:26:42.382251 11914 raft_consensus.cc:2454] T 90e159376c9b41fd8d10a5efd249e817 P bfa7fe045ef14276ae5499bd373f13bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22aa4c1e667043fb877f2ac1acebdc02 in term 0.
I20260620 04:26:42.382697 11999 leader_election.cc:302] T 90e159376c9b41fd8d10a5efd249e817 P 22aa4c1e667043fb877f2ac1acebdc02 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22aa4c1e667043fb877f2ac1acebdc02, bfa7fe045ef14276ae5499bd373f13bb; no voters: cf16724c5e234b0ea15dcefd619c912d
I20260620 04:26:42.382980 12133 log.cc:824] T 90e159376c9b41fd8d10a5efd249e817 P cf16724c5e234b0ea15dcefd619c912d: Log is configured to *not* fsync() on all Append() calls
I20260620 04:26:42.383250 12132 raft_consensus.cc:2781] T 90e159376c9b41fd8d10a5efd249e817 P 22aa4c1e667043fb877f2ac1acebdc02 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 04:26:42.383334 12132 raft_consensus.cc:482] T 90e159376c9b41fd8d10a5efd249e817 P 22aa4c1e667043fb877f2ac1acebdc02 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 04:26:42.383374 12132 raft_consensus.cc:3037] T 90e159376c9b41fd8d10a5efd249e817 P 22aa4c1e667043fb877f2ac1acebdc02 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:26:42.383414 12128 ts_tablet_manager.cc:1428] T 90e159376c9b41fd8d10a5efd249e817 P bfa7fe045ef14276ae5499bd373f13bb: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260620 04:26:42.383751 11975 heartbeater.cc:503] Master 127.0.98.62:38773 was elected leader, sending a full tablet report...
I20260620 04:26:42.384634 12132 raft_consensus.cc:504] T 90e159376c9b41fd8d10a5efd249e817 P 22aa4c1e667043fb877f2ac1acebdc02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22aa4c1e667043fb877f2ac1acebdc02" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40563 } } peers { permanent_uuid: "cf16724c5e234b0ea15dcefd619c912d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45265 } } peers { permanent_uuid: "bfa7fe045ef14276ae5499bd373f13bb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34851 } }
I20260620 04:26:42.385149 12133 tablet_bootstrap.cc:492] T 90e159376c9b41fd8d10a5efd249e817 P cf16724c5e234b0ea15dcefd619c912d: No bootstrap required, opened a new log
I20260620 04:26:42.385255 11791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90e159376c9b41fd8d10a5efd249e817" candidate_uuid: "22aa4c1e667043fb877f2ac1acebdc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf16724c5e234b0ea15dcefd619c912d"
I20260620 04:26:42.385332 12133 ts_tablet_manager.cc:1397] T 90e159376c9b41fd8d10a5efd249e817 P cf16724c5e234b0ea15dcefd619c912d: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.000s
W20260620 04:26:42.385654 11997 leader_election.cc:341] T 90e159376c9b41fd8d10a5efd249e817 P 22aa4c1e667043fb877f2ac1acebdc02 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cf16724c5e234b0ea15dcefd619c912d (127.0.98.1:45265): Illegal state: must be running to vote when last-logged opid is not known
I20260620 04:26:42.386161 12132 leader_election.cc:288] T 90e159376c9b41fd8d10a5efd249e817 P 22aa4c1e667043fb877f2ac1acebdc02 [CANDIDATE]: Term 1 election: Requested vote from peers cf16724c5e234b0ea15dcefd619c912d (127.0.98.1:45265), bfa7fe045ef14276ae5499bd373f13bb (127.0.98.2:34851)
I20260620 04:26:42.386204 11914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90e159376c9b41fd8d10a5efd249e817" candidate_uuid: "22aa4c1e667043fb877f2ac1acebdc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfa7fe045ef14276ae5499bd373f13bb"
I20260620 04:26:42.386299 11914 raft_consensus.cc:3037] T 90e159376c9b41fd8d10a5efd249e817 P bfa7fe045ef14276ae5499bd373f13bb [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:26:42.387204 11914 raft_consensus.cc:2454] T 90e159376c9b41fd8d10a5efd249e817 P bfa7fe045ef14276ae5499bd373f13bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22aa4c1e667043fb877f2ac1acebdc02 in term 1.
I20260620 04:26:42.387444 11999 leader_election.cc:302] T 90e159376c9b41fd8d10a5efd249e817 P 22aa4c1e667043fb877f2ac1acebdc02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22aa4c1e667043fb877f2ac1acebdc02, bfa7fe045ef14276ae5499bd373f13bb; no voters: cf16724c5e234b0ea15dcefd619c912d
I20260620 04:26:42.387692 12132 raft_consensus.cc:2781] T 90e159376c9b41fd8d10a5efd249e817 P 22aa4c1e667043fb877f2ac1acebdc02 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:26:42.387904 12132 raft_consensus.cc:686] T 90e159376c9b41fd8d10a5efd249e817 P 22aa4c1e667043fb877f2ac1acebdc02 [term 1 LEADER]: Becoming Leader. State: Replica: 22aa4c1e667043fb877f2ac1acebdc02, State: Running, Role: LEADER
I20260620 04:26:42.388278 12132 consensus_queue.cc:237] T 90e159376c9b41fd8d10a5efd249e817 P 22aa4c1e667043fb877f2ac1acebdc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22aa4c1e667043fb877f2ac1acebdc02" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40563 } } peers { permanent_uuid: "cf16724c5e234b0ea15dcefd619c912d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45265 } } peers { permanent_uuid: "bfa7fe045ef14276ae5499bd373f13bb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34851 } }
I20260620 04:26:42.388183 12133 raft_consensus.cc:348] T 90e159376c9b41fd8d10a5efd249e817 P cf16724c5e234b0ea15dcefd619c912d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22aa4c1e667043fb877f2ac1acebdc02" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40563 } } peers { permanent_uuid: "cf16724c5e234b0ea15dcefd619c912d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45265 } } peers { permanent_uuid: "bfa7fe045ef14276ae5499bd373f13bb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34851 } }
I20260620 04:26:42.388398 12133 raft_consensus.cc:374] T 90e159376c9b41fd8d10a5efd249e817 P cf16724c5e234b0ea15dcefd619c912d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:26:42.388453 12133 raft_consensus.cc:729] T 90e159376c9b41fd8d10a5efd249e817 P cf16724c5e234b0ea15dcefd619c912d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf16724c5e234b0ea15dcefd619c912d, State: Initialized, Role: FOLLOWER
I20260620 04:26:42.388931 12133 consensus_queue.cc:260] T 90e159376c9b41fd8d10a5efd249e817 P cf16724c5e234b0ea15dcefd619c912d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22aa4c1e667043fb877f2ac1acebdc02" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40563 } } peers { permanent_uuid: "cf16724c5e234b0ea15dcefd619c912d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45265 } } peers { permanent_uuid: "bfa7fe045ef14276ae5499bd373f13bb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34851 } }
I20260620 04:26:42.389902 12133 ts_tablet_manager.cc:1428] T 90e159376c9b41fd8d10a5efd249e817 P cf16724c5e234b0ea15dcefd619c912d: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 04:26:42.390244 11837 heartbeater.cc:503] Master 127.0.98.62:38773 was elected leader, sending a full tablet report...
I20260620 04:26:42.391621 11624 catalog_manager.cc:5495] T 90e159376c9b41fd8d10a5efd249e817 P 22aa4c1e667043fb877f2ac1acebdc02 reported cstate change: term changed from 0 to 1, leader changed from <none> to 22aa4c1e667043fb877f2ac1acebdc02 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "22aa4c1e667043fb877f2ac1acebdc02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22aa4c1e667043fb877f2ac1acebdc02" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40563 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf16724c5e234b0ea15dcefd619c912d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45265 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bfa7fe045ef14276ae5499bd373f13bb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34851 } health_report { overall_health: UNKNOWN } } }
I20260620 04:26:42.405833 12123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:26:42.401353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52062 (local address 127.0.98.3:40563)
0620 04:26:42.401576 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 04:26:42.401581 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:26:42.401594 (+ 13us) server_negotiation.cc:407] Connection header received
0620 04:26:42.401663 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:26:42.401666 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:26:42.401733 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:26:42.401833 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:26:42.402166 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:26:42.402909 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:26:42.404152 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:26:42.404496 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:26:42.404597 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:26:42.404908 (+ 311us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:26:42.405440 (+ 532us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:26:42.405558 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 04:26:42.405636 (+ 78us) server_negotiation.cc:299] Negotiation successful
0620 04:26:42.405694 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142}
I20260620 04:26:42.412319 11791 raft_consensus.cc:3037] T 90e159376c9b41fd8d10a5efd249e817 P cf16724c5e234b0ea15dcefd619c912d [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:26:42.412509 11914 raft_consensus.cc:1261] T 90e159376c9b41fd8d10a5efd249e817 P bfa7fe045ef14276ae5499bd373f13bb [term 1 FOLLOWER]: Refusing update from remote peer 22aa4c1e667043fb877f2ac1acebdc02: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:26:42.413826 11791 raft_consensus.cc:1261] T 90e159376c9b41fd8d10a5efd249e817 P cf16724c5e234b0ea15dcefd619c912d [term 1 FOLLOWER]: Refusing update from remote peer 22aa4c1e667043fb877f2ac1acebdc02: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:26:42.414738 12132 consensus_queue.cc:1035] T 90e159376c9b41fd8d10a5efd249e817 P 22aa4c1e667043fb877f2ac1acebdc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bfa7fe045ef14276ae5499bd373f13bb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:26:42.415248 12142 consensus_queue.cc:1035] T 90e159376c9b41fd8d10a5efd249e817 P 22aa4c1e667043fb877f2ac1acebdc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf16724c5e234b0ea15dcefd619c912d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:26:42.430465 12141 mvcc.cc:205] Tried to move back new op lower bound from 7298783651473158144 to 7298783651385692160. Current Snapshot: MvccSnapshot[applied={T|T < 7298783651473158144}]
I20260620 04:26:42.433538 12144 mvcc.cc:205] Tried to move back new op lower bound from 7298783651473158144 to 7298783651385692160. Current Snapshot: MvccSnapshot[applied={T|T < 7298783651473158144}]
I20260620 04:26:42.442935 11625 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54014:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:26:42.443115 11625 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54014:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:26:42.443138 12143 mvcc.cc:205] Tried to move back new op lower bound from 7298783651473158144 to 7298783651385692160. Current Snapshot: MvccSnapshot[applied={T|T < 7298783651473158144}]
I20260620 04:26:42.445343 11625 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bd9d714c690246bcb7d076d0547c34cb: Sending DeleteTablet for 3 replicas of tablet 90e159376c9b41fd8d10a5efd249e817
I20260620 04:26:42.446023 12043 tablet_service.cc:1514] Processing DeleteTablet for tablet 90e159376c9b41fd8d10a5efd249e817 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:26:42 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52046
I20260620 04:26:42.446081 11760 tablet_service.cc:1514] Processing DeleteTablet for tablet 90e159376c9b41fd8d10a5efd249e817 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:26:42 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50450
I20260620 04:26:42.446439 12152 tablet_replica.cc:330] T 90e159376c9b41fd8d10a5efd249e817 P 22aa4c1e667043fb877f2ac1acebdc02: stopping tablet replica
I20260620 04:26:42.446579 11909 tablet_service.cc:1514] Processing DeleteTablet for tablet 90e159376c9b41fd8d10a5efd249e817 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:26:42 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43236
I20260620 04:26:42.446852 12152 raft_consensus.cc:2229] T 90e159376c9b41fd8d10a5efd249e817 P 22aa4c1e667043fb877f2ac1acebdc02 [term 1 LEADER]: Raft consensus shutting down.
I20260620 04:26:42.447284 12152 raft_consensus.cc:2258] T 90e159376c9b41fd8d10a5efd249e817 P 22aa4c1e667043fb877f2ac1acebdc02 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:26:42.447741 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 11706
I20260620 04:26:42.449506 12152 ts_tablet_manager.cc:1905] T 90e159376c9b41fd8d10a5efd249e817 P 22aa4c1e667043fb877f2ac1acebdc02: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:26:42.452246 12152 ts_tablet_manager.cc:1918] T 90e159376c9b41fd8d10a5efd249e817 P 22aa4c1e667043fb877f2ac1acebdc02: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:26:42.452339 12152 log.cc:1196] T 90e159376c9b41fd8d10a5efd249e817 P 22aa4c1e667043fb877f2ac1acebdc02: Deleting WAL directory at /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/minicluster-data/ts-2/wal/wals/90e159376c9b41fd8d10a5efd249e817
I20260620 04:26:42.452656 12152 ts_tablet_manager.cc:1939] T 90e159376c9b41fd8d10a5efd249e817 P 22aa4c1e667043fb877f2ac1acebdc02: Deleting consensus metadata
I20260620 04:26:42.453778 11611 catalog_manager.cc:4841] TS 22aa4c1e667043fb877f2ac1acebdc02 (127.0.98.3:40563): tablet 90e159376c9b41fd8d10a5efd249e817 (table test-table [id=27088cf1ab614d9087ae25a8dda470e6]) successfully deleted
I20260620 04:26:42.456305 12154 tablet_replica.cc:330] T 90e159376c9b41fd8d10a5efd249e817 P bfa7fe045ef14276ae5499bd373f13bb: stopping tablet replica
I20260620 04:26:42.456642 12154 raft_consensus.cc:2229] T 90e159376c9b41fd8d10a5efd249e817 P bfa7fe045ef14276ae5499bd373f13bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:26:42.456972 12154 raft_consensus.cc:2258] T 90e159376c9b41fd8d10a5efd249e817 P bfa7fe045ef14276ae5499bd373f13bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:26:42.461346 12154 ts_tablet_manager.cc:1905] T 90e159376c9b41fd8d10a5efd249e817 P bfa7fe045ef14276ae5499bd373f13bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:26:42.464020 12154 ts_tablet_manager.cc:1918] T 90e159376c9b41fd8d10a5efd249e817 P bfa7fe045ef14276ae5499bd373f13bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:26:42.464118 12154 log.cc:1196] T 90e159376c9b41fd8d10a5efd249e817 P bfa7fe045ef14276ae5499bd373f13bb: Deleting WAL directory at /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929496330398-392-0/minicluster-data/ts-1/wal/wals/90e159376c9b41fd8d10a5efd249e817
I20260620 04:26:42.464404 12154 ts_tablet_manager.cc:1939] T 90e159376c9b41fd8d10a5efd249e817 P bfa7fe045ef14276ae5499bd373f13bb: Deleting consensus metadata
I20260620 04:26:42.465878 11612 catalog_manager.cc:4841] TS bfa7fe045ef14276ae5499bd373f13bb (127.0.98.2:34851): tablet 90e159376c9b41fd8d10a5efd249e817 (table test-table [id=27088cf1ab614d9087ae25a8dda470e6]) successfully deleted
W20260620 04:26:42.469034 11610 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45265 (error 108)
I20260620 04:26:42.470122 12122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:26:42.469804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45265 (local address 127.0.0.1:50454)
0620 04:26:42.469979 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 04:26:42.470041 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
W20260620 04:26:42.470383 11610 catalog_manager.cc:4584] TS cf16724c5e234b0ea15dcefd619c912d (127.0.98.1:45265): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 90e159376c9b41fd8d10a5efd249e817: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45265: connect: Connection refused (error 111)
I20260620 04:26:42.480185 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 11845
I20260620 04:26:42.486212 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 11980
Jun 20 04:26:42 dist-test-slave-nk6d krb5kdc[11149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929601, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 04:26:42.492834 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 11593
I20260620 04:26:42.499817 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 04:26:42.844262 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 04:26:42.844419 392 mini_ranger.cc:66] Stopping Ranger...
I20260620 04:26:43.178768 392 mini_ranger.cc:68] Stopped Ranger
2026-06-20 04:26:43.179 UTC [11179] LOG: received smart shutdown request
2026-06-20 04:26:43.182 UTC [11179] LOG: background worker "logical replication launcher" (PID 11188) exited with exit code 1
2026-06-20 04:26:43.182 UTC [11183] LOG: shutting down
2026-06-20 04:26:43.317 UTC [11179] LOG: database system is shut down
2026-06-20T04:26:43Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (43970 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:26:43 dist-test-slave-nk6d krb5kdc[12165](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:26:43 dist-test-slave-nk6d krb5kdc[12165](info): set up 2 sockets
Jun 20 04:26:43 dist-test-slave-nk6d krb5kdc[12165](info): commencing operation
krb5kdc: starting...
W20260620 04:26:44.355867 392 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/krb5kdc/test-user.keytab.
Jun 20 04:26:44 dist-test-slave-nk6d krb5kdc[12165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929604, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:26:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:26:44Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260620 04:26:44.506500 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taske80qNG/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/postgres -l logfile start
2026-06-20 04:26:46.701 UTC [12195] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 04:26:46.701 UTC [12195] LOG: listening on IPv4 address "127.0.98.20", port 52767
2026-06-20 04:26:46.704 UTC [12195] LOG: listening on Unix socket "/tmp/.s.PGSQL.52767"
2026-06-20 04:26:46.732 UTC [12198] LOG: database system was shut down at 2026-06-20 04:26:45 UTC
2026-06-20 04:26:46.736 UTC [12195] LOG: database system is ready to accept connections
I20260620 04:26:47.607533 392 mini_postgres.cc:96] Postgres bound to 52767
2026-06-20 04:26:47.613 UTC [12206] FATAL: database "slave" does not exist
127.0.98.20:52767 - accepting connections
I20260620 04:26:47.613684 392 mini_ranger.cc:161] Starting Ranger...
I20260620 04:26:47.627297 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 04:26:48.033242 392 mini_ranger.cc:87] Created ranger Postgres database
I20260620 04:26:48.033325 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/ranger-admin
2026-06-20 04:26:48,298 [I] DB FLAVOR :POSTGRES
2026-06-20 04:26:48,298 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 04:26:48,298 [I] Checking connection..
2026-06-20 04:26:48,298 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske80qNG/build/debug/bin/postgresql.jar:/tmp/dist-test-taske80qNG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52767/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 04:26:48,530 [I] Checking connection passed.
2026-06-20 04:26:48,530 [I] --------- Verifying version history table ---------
2026-06-20 04:26:48,530 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske80qNG/build/debug/bin/postgresql.jar:/tmp/dist-test-taske80qNG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52767/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 04:26:48,786 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 04:26:48,787 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske80qNG/build/debug/bin/postgresql.jar:/tmp/dist-test-taske80qNG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52767/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 04:26:49,039 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 04:26:49,039 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 04:26:49,039 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske80qNG/build/debug/bin/postgresql.jar:/tmp/dist-test-taske80qNG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52767/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taske80qNG/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 04:26:49.267 UTC [12270] WARNING: there is no transaction in progress
2026-06-20 04:26:49,272 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 04:26:49,272 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske80qNG/build/debug/bin/postgresql.jar:/tmp/dist-test-taske80qNG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52767/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 04:26:49,525 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 04:26:49,525 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 04:26:49,525 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske80qNG/build/debug/bin/postgresql.jar:/tmp/dist-test-taske80qNG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52767/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 04:26:49,756 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske80qNG/build/debug/bin/postgresql.jar:/tmp/dist-test-taske80qNG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52767/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 04:26:50,004 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske80qNG/build/debug/bin/postgresql.jar:/tmp/dist-test-taske80qNG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52767/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-nk6d.c.gcp-upstream.internal','N') ;"
2026-06-20 04:26:50,246 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske80qNG/build/debug/bin/postgresql.jar:/tmp/dist-test-taske80qNG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52767/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 04:26:50,476 [I] Table x_portal_user does not exist in database ranger
2026-06-20 04:26:50,476 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske80qNG/build/debug/bin/postgresql.jar:/tmp/dist-test-taske80qNG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52767/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 04:26:50,725 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 04:26:50,726 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske80qNG/build/debug/bin/postgresql.jar:/tmp/dist-test-taske80qNG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52767/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 04:26:50,969 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske80qNG/build/debug/bin/postgresql.jar:/tmp/dist-test-taske80qNG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52767/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 04:26:51,200 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 04:26:51,201 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske80qNG/build/debug/bin/postgresql.jar:/tmp/dist-test-taske80qNG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52767/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taske80qNG/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 04:26:52.087 UTC [12378] WARNING: there is no transaction in progress
2026-06-20 04:26:52.101 UTC [12378] WARNING: there is no transaction in progress
2026-06-20 04:26:52.116 UTC [12378] WARNING: there is no transaction in progress
2026-06-20 04:26:52.131 UTC [12378] WARNING: there is no transaction in progress
2026-06-20 04:26:52.151 UTC [12378] WARNING: there is no transaction in progress
2026-06-20 04:26:52.171 UTC [12378] WARNING: there is no transaction in progress
2026-06-20 04:26:52.233 UTC [12378] WARNING: there is no transaction in progress
2026-06-20 04:26:52.240 UTC [12378] WARNING: there is no transaction in progress
2026-06-20 04:26:52.253 UTC [12378] WARNING: there is no transaction in progress
2026-06-20 04:26:52.263 UTC [12378] WARNING: there is no transaction in progress
2026-06-20 04:26:52.272 UTC [12378] WARNING: there is no transaction in progress
2026-06-20 04:26:52.284 UTC [12378] WARNING: there is no transaction in progress
2026-06-20 04:26:52.293 UTC [12378] WARNING: there is no transaction in progress
2026-06-20 04:26:52.299 UTC [12378] WARNING: there is no transaction in progress
2026-06-20 04:26:52.305 UTC [12378] WARNING: there is no transaction in progress
2026-06-20 04:26:52.311 UTC [12378] WARNING: there is no transaction in progress
2026-06-20 04:26:52,886 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 04:26:52,887 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske80qNG/build/debug/bin/postgresql.jar:/tmp/dist-test-taske80qNG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52767/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 04:26:53,132 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 04:26:53,132 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske80qNG/build/debug/bin/postgresql.jar:/tmp/dist-test-taske80qNG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52767/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 04:26:53,371 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 04:26:53,371 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske80qNG/build/debug/bin/postgresql.jar:/tmp/dist-test-taske80qNG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52767/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 04:26:53,624 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 04:26:53,625 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske80qNG/build/debug/bin/postgresql.jar:/tmp/dist-test-taske80qNG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52767/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 04:26:53,861 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske80qNG/build/debug/bin/postgresql.jar:/tmp/dist-test-taske80qNG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52767/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 04:26:54,100 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske80qNG/build/debug/bin/postgresql.jar:/tmp/dist-test-taske80qNG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52767/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-nk6d.c.gcp-upstream.internal';"
2026-06-20 04:26:54,342 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 04:26:54,342 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske80qNG/build/debug/bin/postgresql.jar:/tmp/dist-test-taske80qNG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52767/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 04:26:54,582 [I] DB_PATCHES have already been applied
I20260620 04:26:54.587344 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taske80qNG/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taske80qNG/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taske80qNG/build/debug/bin/postgresql.jar:/tmp/dist-test-taske80qNG/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 04:26:54.587404 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260620 04:26:54.590392 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:55437
Jun 20, 2026 4:26:54 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 4:26:54 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 4:26:55 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 4:26:55 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 4:26:55 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 4:26:55 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 4:26:55 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 04:26:55.582091 392 mini_ranger.cc:160] Time spent starting Ranger: real 7.968s user 0.000s sys 0.008s
Jun 20, 2026 4:27:15 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 4:27:15 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 20, 2026 4:27:15 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 4:27:15 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 4:27:15 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 4:27:15 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 4:27:15 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 4:27:15 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 4:27:15 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 4:27:15 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 4:27:15 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 4:27:15 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 4:27:15 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 4:27:15 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 4:27:15 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 4:27:15 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 4:27:15 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 4:27:16 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 04:27:16.793673 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 04:27:16 dist-test-slave-nk6d krb5kdc[12165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929636, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 04:27:16.885799 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 04:27:16.895565 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 04:27:17.617498 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 04:27:17.617566 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/ranger-kms
I20260620 04:27:17.617579 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:52767
2026-06-20 04:27:17,766 [I] DB FLAVOR :POSTGRES
2026-06-20 04:27:17,766 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 04:27:17,766 [I] Checking connection
2026-06-20 04:27:17,766 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske80qNG/build/debug/bin/postgresql.jar:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52767/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 04:27:17,995 [I] connection success
2026-06-20 04:27:17,995 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 04:27:17,995 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 04:27:17,996 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske80qNG/build/debug/bin/postgresql.jar:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52767/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 04:27:18,217 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 04:27:18,217 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 04:27:18,218 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 04:27:18,218 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske80qNG/build/debug/bin/postgresql.jar:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52767/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 04:27:18,491 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 04:27:18.969512 392 mini_ranger_kms.cc:310] Created kms service
I20260620 04:27:19.163750 392 mini_ranger_kms.cc:326] Created kudu user
I20260620 04:27:19.204051 392 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 04:27:19 dist-test-slave-nk6d krb5kdc[12165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929636, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 04:27:19.581599 392 mini_ranger_kms.cc:384] Added ranger policy
I20260620 04:27:19.581749 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taske80qNG/build/debug/bin/postgresql.jar:/tmp/dist-test-taske80qNG/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taske80qNG/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 04:27:19.581770 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260620 04:27:19.585458 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12558
I20260620 04:27:19.585526 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:44373
Jun 20, 2026 4:27:20 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 4:27:20 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 4:27:20 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 4:27:20 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 4:27:21 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 4:27:21 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260620 04:27:21.615865 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.730s user 0.010s sys 0.123s
I20260620 04:27:21.616057 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 04:27:21.616145 392 mini_ranger_kms.cc:401] 127.0.98.20:44373/kms/v1/keys
Jun 20 04:27:21 dist-test-slave-nk6d krb5kdc[12165](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781929641, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 04:27:22 dist-test-slave-nk6d krb5kdc[12165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929641, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 20 04:27:22 dist-test-slave-nk6d krb5kdc[12165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929641, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 20 04:27:24 dist-test-slave-nk6d krb5kdc[12165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929644, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 04:27:24.718955 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46301
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46301
--ranger_config_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:44373/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:27:24.826984 12609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:27:24.827260 12609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:27:24.827315 12609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:27:24.831385 12609 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:27:24.831452 12609 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:27:24.831478 12609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:27:24.831499 12609 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:27:24.831524 12609 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:27:24.837038 12609 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:44373/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46301
--ranger_config_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46301
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12609
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:27:24.838057 12609 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:27:24.839063 12609 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:27:24.848824 12615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:27:24.848878 12609 server_base.cc:1034] running on GCE node
W20260620 04:27:24.849068 12617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:27:24.848831 12614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:27:24.849797 12609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:27:24.851193 12609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:27:24.852376 12609 hybrid_clock.cc:648] HybridClock initialized: now 1781929644852346 us; error 40 us; skew 500 ppm
Jun 20 04:27:24 dist-test-slave-nk6d krb5kdc[12165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929644, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:27:24.855580 12609 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 04:27:24.856510 12609 webserver.cc:460] Webserver started at http://127.0.98.62:40161/ using document root <none> and password file <none>
I20260620 04:27:24.857016 12609 fs_manager.cc:359] Metadata directory not provided
I20260620 04:27:24.857066 12609 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:27:24.857257 12609 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 04:27:24 dist-test-slave-nk6d krb5kdc[12165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929644, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 04:27:24.905428 12609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/master-0/data/instance:
uuid: "b8437fed23e6424095fee92cfb17a227"
format_stamp: "Formatted at 2026-06-20 04:27:24 on dist-test-slave-nk6d"
server_key: "60889391fc3fbeadb3a2ddf761024ac5"
server_key_iv: "fc0cbbe599e5ae774a1fa7f493356a1d"
server_key_version: "kuduclusterkey@0"
I20260620 04:27:24.906054 12609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/master-0/wal/instance:
uuid: "b8437fed23e6424095fee92cfb17a227"
format_stamp: "Formatted at 2026-06-20 04:27:24 on dist-test-slave-nk6d"
server_key: "60889391fc3fbeadb3a2ddf761024ac5"
server_key_iv: "fc0cbbe599e5ae774a1fa7f493356a1d"
server_key_version: "kuduclusterkey@0"
I20260620 04:27:24.909814 12609 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 04:27:24.924762 12624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:27:24.926309 12609 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 04:27:24.926467 12609 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/master-0/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/master-0/wal
uuid: "b8437fed23e6424095fee92cfb17a227"
format_stamp: "Formatted at 2026-06-20 04:27:24 on dist-test-slave-nk6d"
server_key: "60889391fc3fbeadb3a2ddf761024ac5"
server_key_iv: "fc0cbbe599e5ae774a1fa7f493356a1d"
server_key_version: "kuduclusterkey@0"
I20260620 04:27:24.926590 12609 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:27:24.938087 12609 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:27:24.940861 12609 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:27:24.941040 12609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:27:24.948050 12609 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46301
I20260620 04:27:24.948087 12676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46301 every 8 connection(s)
I20260620 04:27:24.948992 12609 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:27:24.951637 12677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:27:24.956192 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 12609
I20260620 04:27:24.956328 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/master-0/wal/instance
I20260620 04:27:24.956633 392 external_mini_cluster.cc:1428] Setting key 4aa2b9bbd61594879988f7dd4b2860ef
I20260620 04:27:24.957674 12677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8437fed23e6424095fee92cfb17a227: Bootstrap starting.
I20260620 04:27:24.960170 12677 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b8437fed23e6424095fee92cfb17a227: Neither blocks nor log segments found. Creating new log.
I20260620 04:27:24.961119 12677 log.cc:824] T 00000000000000000000000000000000 P b8437fed23e6424095fee92cfb17a227: Log is configured to *not* fsync() on all Append() calls
I20260620 04:27:24.963034 12677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8437fed23e6424095fee92cfb17a227: No bootstrap required, opened a new log
Jun 20 04:27:24 dist-test-slave-nk6d krb5kdc[12165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929644, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:27:24.967310 12677 raft_consensus.cc:348] T 00000000000000000000000000000000 P b8437fed23e6424095fee92cfb17a227 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8437fed23e6424095fee92cfb17a227" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46301 } }
I20260620 04:27:24.967638 12677 raft_consensus.cc:374] T 00000000000000000000000000000000 P b8437fed23e6424095fee92cfb17a227 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:27:24.967723 12677 raft_consensus.cc:729] T 00000000000000000000000000000000 P b8437fed23e6424095fee92cfb17a227 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8437fed23e6424095fee92cfb17a227, State: Initialized, Role: FOLLOWER
I20260620 04:27:24.968246 12677 consensus_queue.cc:260] T 00000000000000000000000000000000 P b8437fed23e6424095fee92cfb17a227 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8437fed23e6424095fee92cfb17a227" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46301 } }
I20260620 04:27:24.968412 12677 raft_consensus.cc:388] T 00000000000000000000000000000000 P b8437fed23e6424095fee92cfb17a227 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:27:24.968500 12677 raft_consensus.cc:482] T 00000000000000000000000000000000 P b8437fed23e6424095fee92cfb17a227 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:27:24.968606 12677 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b8437fed23e6424095fee92cfb17a227 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:27:24.969561 12677 raft_consensus.cc:504] T 00000000000000000000000000000000 P b8437fed23e6424095fee92cfb17a227 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8437fed23e6424095fee92cfb17a227" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46301 } }
I20260620 04:27:24.969904 12677 leader_election.cc:302] T 00000000000000000000000000000000 P b8437fed23e6424095fee92cfb17a227 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b8437fed23e6424095fee92cfb17a227; no voters:
I20260620 04:27:24.970305 12677 leader_election.cc:288] T 00000000000000000000000000000000 P b8437fed23e6424095fee92cfb17a227 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:27:24.970679 12682 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b8437fed23e6424095fee92cfb17a227 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:27:24.971657 12677 sys_catalog.cc:564] T 00000000000000000000000000000000 P b8437fed23e6424095fee92cfb17a227 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:27:24.973075 12680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:24.958158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49980 (local address 127.0.98.62:46301)
0620 04:27:24.958675 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0620 04:27:24.958695 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 04:27:24.958740 (+ 45us) server_negotiation.cc:407] Connection header received
0620 04:27:24.959608 (+ 868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:27:24.959635 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:27:24.960129 (+ 494us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:27:24.960509 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:27:24.961390 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:24.962857 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:24.964019 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:24.964705 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:24.964853 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:27:24.967872 (+ 3019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:27:24.967880 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:27:24.967898 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:27:24.967931 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:27:24.969884 (+ 1953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:27:24.970366 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:27:24.970372 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:27:24.970378 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:27:24.970508 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:27:24.970809 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:27:24.970812 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:27:24.970813 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:27:24.971337 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:27:24.971517 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0620 04:27:24.971790 (+ 273us) server_negotiation.cc:299] Negotiation successful
0620 04:27:24.972117 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":140,"threads_started":1}
I20260620 04:27:24.973527 12682 raft_consensus.cc:686] T 00000000000000000000000000000000 P b8437fed23e6424095fee92cfb17a227 [term 1 LEADER]: Becoming Leader. State: Replica: b8437fed23e6424095fee92cfb17a227, State: Running, Role: LEADER
I20260620 04:27:24.973861 12682 consensus_queue.cc:237] T 00000000000000000000000000000000 P b8437fed23e6424095fee92cfb17a227 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8437fed23e6424095fee92cfb17a227" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46301 } }
I20260620 04:27:24.975478 12683 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8437fed23e6424095fee92cfb17a227 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b8437fed23e6424095fee92cfb17a227" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8437fed23e6424095fee92cfb17a227" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46301 } } }
I20260620 04:27:24.975593 12683 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8437fed23e6424095fee92cfb17a227 [sys.catalog]: This master's current role is: LEADER
I20260620 04:27:24.976397 12677 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 04:27:24.976727 12694 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8437fed23e6424095fee92cfb17a227 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8437fed23e6424095fee92cfb17a227. Latest consensus state: current_term: 1 leader_uuid: "b8437fed23e6424095fee92cfb17a227" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8437fed23e6424095fee92cfb17a227" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46301 } } }
I20260620 04:27:24.976899 12694 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8437fed23e6424095fee92cfb17a227 [sys.catalog]: This master's current role is: LEADER
I20260620 04:27:24.977116 12696 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:27:24.982589 12696 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:27:24.990305 12696 catalog_manager.cc:1347] Generated new cluster ID: 55b11160e31b49e9b82ea4776ffa7dab
I20260620 04:27:24.990378 12696 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:27:25.017009 12696 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:27:25.018812 12696 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:27:25.026293 12696 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b8437fed23e6424095fee92cfb17a227: Generated new TSK 0
I20260620 04:27:25.027258 12696 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 04:27:25.746788 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:44373/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46301
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:27:25.879762 12722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:27:25.880214 12722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:27:25.880326 12722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:27:25.885504 12722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:27:25.885571 12722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:27:25.885634 12722 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 04:27:25.893102 12722 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:44373/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46301
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12722
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:27:25.894472 12722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:27:25.895764 12722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:27:25.907137 12730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:27:25.907153 12732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:27:25.907732 12729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:27:25.908071 12722 server_base.cc:1034] running on GCE node
I20260620 04:27:25.908617 12722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:27:25.909387 12722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:27:25.910573 12722 hybrid_clock.cc:648] HybridClock initialized: now 1781929645910542 us; error 45 us; skew 500 ppm
Jun 20 04:27:25 dist-test-slave-nk6d krb5kdc[12165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929645, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:27:25.914069 12722 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 04:27:25.915127 12722 webserver.cc:460] Webserver started at http://127.0.98.1:35253/ using document root <none> and password file <none>
I20260620 04:27:25.915784 12722 fs_manager.cc:359] Metadata directory not provided
I20260620 04:27:25.915867 12722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:27:25.916100 12722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 04:27:25 dist-test-slave-nk6d krb5kdc[12165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929645, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 04:27:25.935125 12722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/ts-0/data/instance:
uuid: "0e23e8c323e54928b36dd233b1ee1d97"
format_stamp: "Formatted at 2026-06-20 04:27:25 on dist-test-slave-nk6d"
server_key: "5811f44c7685eb998833d374ffd839ee"
server_key_iv: "eabb94a43f9afaf3c7b8bfe6317c5929"
server_key_version: "kuduclusterkey@0"
I20260620 04:27:25.936012 12722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/ts-0/wal/instance:
uuid: "0e23e8c323e54928b36dd233b1ee1d97"
format_stamp: "Formatted at 2026-06-20 04:27:25 on dist-test-slave-nk6d"
server_key: "5811f44c7685eb998833d374ffd839ee"
server_key_iv: "eabb94a43f9afaf3c7b8bfe6317c5929"
server_key_version: "kuduclusterkey@0"
I20260620 04:27:25.941344 12722 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.003s
I20260620 04:27:25.962975 12739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:27:25.968760 12722 fs_manager.cc:725] Time spent opening block manager: real 0.013s user 0.003s sys 0.000s
I20260620 04:27:25.968906 12722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/ts-0/wal
uuid: "0e23e8c323e54928b36dd233b1ee1d97"
format_stamp: "Formatted at 2026-06-20 04:27:25 on dist-test-slave-nk6d"
server_key: "5811f44c7685eb998833d374ffd839ee"
server_key_iv: "eabb94a43f9afaf3c7b8bfe6317c5929"
server_key_version: "kuduclusterkey@0"
I20260620 04:27:25.969038 12722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:27:25.993235 12722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:27:25.998976 12722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:27:25.999240 12722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:27:26.000092 12722 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:27:26.001251 12722 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:27:26.001363 12722 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:27:26.001495 12722 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:27:26.001580 12722 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:27:26.011792 12722 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33171
I20260620 04:27:26.011983 12854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33171 every 8 connection(s)
I20260620 04:27:26.012806 12722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/ts-0/data/info.pb
I20260620 04:27:26.018370 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 12722
I20260620 04:27:26.018481 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/ts-0/wal/instance
I20260620 04:27:26.018777 392 external_mini_cluster.cc:1428] Setting key 723bde665cafc1b3a219f95ed5f213c4
Jun 20 04:27:26 dist-test-slave-nk6d krb5kdc[12165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929645, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:27:26.028556 12857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:26.014565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41713 (local address 127.0.98.62:46301)
0620 04:27:26.014845 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0620 04:27:26.014853 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 04:27:26.015643 (+ 790us) server_negotiation.cc:407] Connection header received
0620 04:27:26.016683 (+ 1040us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:27:26.016687 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:27:26.016746 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:27:26.016824 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:27:26.017879 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:26.018558 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:26.020730 (+ 2172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:26.021023 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:26.021095 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:27:26.024082 (+ 2987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:27:26.024086 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:27:26.024088 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:27:26.024113 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:27:26.025914 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:27:26.026619 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:27:26.026623 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:27:26.026624 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:27:26.026719 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:27:26.027137 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:27:26.027141 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:27:26.027143 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:27:26.027370 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:27:26.027453 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 04:27:26.028231 (+ 778us) server_negotiation.cc:299] Negotiation successful
0620 04:27:26.028360 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":101,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 04:27:26.030583 12858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:26.015043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46301 (local address 127.0.98.1:41713)
0620 04:27:26.015487 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:26.015539 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 04:27:26.016468 (+ 929us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:27:26.017001 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:27:26.017008 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:27:26.017271 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:27:26.017741 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:27:26.017754 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:27:26.018717 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:27:26.018721 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:27:26.020589 (+ 1868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:27:26.020601 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:27:26.021306 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:27:26.021310 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:27:26.021496 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:27:26.022145 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:27:26.022169 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:27:26.023867 (+ 1698us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:27:26.026097 (+ 2230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:27:26.026103 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:27:26.026134 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:27:26.026496 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:27:26.026827 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:27:26.026831 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:27:26.026833 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:27:26.027015 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:27:26.027523 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:27:26.027530 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:27:26.027889 (+ 359us) client_negotiation.cc:769] Sending connection context
0620 04:27:26.029116 (+ 1227us) client_negotiation.cc:240] Negotiation successful
0620 04:27:26.029486 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":99,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 04:27:26.032181 12855 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46301
I20260620 04:27:26.032431 12855 heartbeater.cc:464] Registering TS with master...
I20260620 04:27:26.032992 12855 heartbeater.cc:511] Master 127.0.98.62:46301 requested a full tablet report, sending...
I20260620 04:27:26.034652 12641 ts_manager.cc:194] Registered new tserver with Master: 0e23e8c323e54928b36dd233b1ee1d97 (127.0.98.1:33171)
I20260620 04:27:26.035755 12641 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:41713
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 04:27:26.048930 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:44373/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46301
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 04:27:26 dist-test-slave-nk6d krb5kdc[12165](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781929646, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 04:27:26.175872 12861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:27:26.176201 12861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:27:26.176262 12861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:27:26.180409 12861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:27:26.180472 12861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:27:26.180552 12861 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 04:27:26.186036 12861 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:44373/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46301
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12861
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:27:26.187240 12861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:27:26.188320 12861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:27:26.197804 12870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:27:26.198056 12868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:27:26.197804 12867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:27:26.198374 12861 server_base.cc:1034] running on GCE node
I20260620 04:27:26.198812 12861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:27:26.199450 12861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:27:26.200610 12861 hybrid_clock.cc:648] HybridClock initialized: now 1781929646200570 us; error 54 us; skew 500 ppm
Jun 20 04:27:26 dist-test-slave-nk6d krb5kdc[12165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929646, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:27:26.203205 12861 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 04:27:26.204100 12861 webserver.cc:460] Webserver started at http://127.0.98.2:32779/ using document root <none> and password file <none>
I20260620 04:27:26.204627 12861 fs_manager.cc:359] Metadata directory not provided
I20260620 04:27:26.204698 12861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:27:26.204922 12861 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 04:27:26 dist-test-slave-nk6d krb5kdc[12165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929646, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 04:27:26.217819 12861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/ts-1/data/instance:
uuid: "3d6563b6a3a7478e99161bb8178ef39c"
format_stamp: "Formatted at 2026-06-20 04:27:26 on dist-test-slave-nk6d"
server_key: "72c290c7d175efc98c1514d603b2c652"
server_key_iv: "851f4ca665e3af6bd87006ffb31f3f66"
server_key_version: "kuduclusterkey@0"
I20260620 04:27:26.218865 12861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/ts-1/wal/instance:
uuid: "3d6563b6a3a7478e99161bb8178ef39c"
format_stamp: "Formatted at 2026-06-20 04:27:26 on dist-test-slave-nk6d"
server_key: "72c290c7d175efc98c1514d603b2c652"
server_key_iv: "851f4ca665e3af6bd87006ffb31f3f66"
server_key_version: "kuduclusterkey@0"
I20260620 04:27:26.225503 12861 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260620 04:27:26.262418 12877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:27:26.264205 12861 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260620 04:27:26.264335 12861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/ts-1/wal
uuid: "3d6563b6a3a7478e99161bb8178ef39c"
format_stamp: "Formatted at 2026-06-20 04:27:26 on dist-test-slave-nk6d"
server_key: "72c290c7d175efc98c1514d603b2c652"
server_key_iv: "851f4ca665e3af6bd87006ffb31f3f66"
server_key_version: "kuduclusterkey@0"
I20260620 04:27:26.264462 12861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:27:26.294723 12861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:27:26.306576 12861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:27:26.311761 12861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:27:26.312769 12861 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:27:26.313937 12861 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:27:26.318117 12861 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260620 04:27:26.318284 12861 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:27:26.318360 12861 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:27:26.332316 12861 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42783
I20260620 04:27:26.332357 12990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42783 every 8 connection(s)
I20260620 04:27:26.333336 12861 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/ts-1/data/info.pb
I20260620 04:27:26.337320 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 12861
I20260620 04:27:26.337445 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/ts-1/wal/instance
I20260620 04:27:26.337760 392 external_mini_cluster.cc:1428] Setting key 58e8baedfb5fc5e3a63f3efc2998ec78
Jun 20 04:27:26 dist-test-slave-nk6d krb5kdc[12165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929646, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:27:26.349615 12857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:26.335298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49303 (local address 127.0.98.62:46301)
0620 04:27:26.335427 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 04:27:26.335431 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:27:26.336481 (+ 1050us) server_negotiation.cc:407] Connection header received
0620 04:27:26.337761 (+ 1280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:27:26.337766 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:27:26.337821 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:27:26.337876 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:27:26.339486 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:26.340444 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:26.342061 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:26.342365 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:26.342435 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:27:26.345347 (+ 2912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:27:26.345351 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:27:26.345354 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:27:26.345384 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:27:26.347083 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:27:26.347714 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:27:26.347718 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:27:26.347720 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:27:26.347809 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:27:26.348253 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:27:26.348257 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:27:26.348259 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:27:26.348485 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:27:26.348560 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 04:27:26.349309 (+ 749us) server_negotiation.cc:299] Negotiation successful
0620 04:27:26.349445 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 04:27:26.351166 12993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:26.335758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46301 (local address 127.0.98.2:49303)
0620 04:27:26.336306 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:26.336375 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0620 04:27:26.337502 (+ 1127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:27:26.338290 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:27:26.338297 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:27:26.338636 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:27:26.339322 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:27:26.339343 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:27:26.340613 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:27:26.340616 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:27:26.341922 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:27:26.341933 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:27:26.342621 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:27:26.342625 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:27:26.342824 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:27:26.343539 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:27:26.343560 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:27:26.345158 (+ 1598us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:27:26.347232 (+ 2074us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:27:26.347239 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:27:26.347257 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:27:26.347600 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:27:26.347920 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:27:26.347923 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:27:26.347925 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:27:26.348143 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:27:26.348641 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:27:26.348649 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:27:26.349005 (+ 356us) client_negotiation.cc:769] Sending connection context
0620 04:27:26.349847 (+ 842us) client_negotiation.cc:240] Negotiation successful
0620 04:27:26.350206 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":151,"threads_started":1}
I20260620 04:27:26.352442 12991 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46301
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260620 04:27:26.352706 12991 heartbeater.cc:464] Registering TS with master...
I20260620 04:27:26.353260 12991 heartbeater.cc:511] Master 127.0.98.62:46301 requested a full tablet report, sending...
I20260620 04:27:26.354298 12641 ts_manager.cc:194] Registered new tserver with Master: 3d6563b6a3a7478e99161bb8178ef39c (127.0.98.2:42783)
I20260620 04:27:26.354960 12641 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:49303
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 04:27:26.375444 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:44373/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46301
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:27:26.546228 12996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:27:26.546693 12996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:27:26.546805 12996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:27:26.552573 12996 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:27:26.552708 12996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:27:26.552829 12996 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 04:27:26.560619 12996 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:44373/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46301
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12996
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:27:26.562235 12996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:27:26.563607 12996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:27:26.576679 13002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:27:26.577221 13001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:27:26.578498 12996 server_base.cc:1034] running on GCE node
W20260620 04:27:26.577316 13004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:27:26.581089 12996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:27:26.581866 12996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:27:26.583029 12996 hybrid_clock.cc:648] HybridClock initialized: now 1781929646582962 us; error 83 us; skew 500 ppm
Jun 20 04:27:26 dist-test-slave-nk6d krb5kdc[12165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929646, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:27:26.586436 12996 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 04:27:26.587494 12996 webserver.cc:460] Webserver started at http://127.0.98.3:42257/ using document root <none> and password file <none>
I20260620 04:27:26.588918 12996 fs_manager.cc:359] Metadata directory not provided
I20260620 04:27:26.589028 12996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:27:26.589294 12996 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 04:27:26 dist-test-slave-nk6d krb5kdc[12165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929646, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 04:27:26.611091 12996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/ts-2/data/instance:
uuid: "0a704308e03e4c2f83672bbb9f35ff79"
format_stamp: "Formatted at 2026-06-20 04:27:26 on dist-test-slave-nk6d"
server_key: "7c55a2a362ce5609a3352f0148609f11"
server_key_iv: "c6fb2e057d342f72b2285fba6e48672c"
server_key_version: "kuduclusterkey@0"
I20260620 04:27:26.612402 12996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/ts-2/wal/instance:
uuid: "0a704308e03e4c2f83672bbb9f35ff79"
format_stamp: "Formatted at 2026-06-20 04:27:26 on dist-test-slave-nk6d"
server_key: "7c55a2a362ce5609a3352f0148609f11"
server_key_iv: "c6fb2e057d342f72b2285fba6e48672c"
server_key_version: "kuduclusterkey@0"
Jun 20 04:27:26 dist-test-slave-nk6d krb5kdc[12165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929646, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 04:27:26.626158 12996 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.004s sys 0.000s
I20260620 04:27:26.675938 13011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:27:26.680836 12996 fs_manager.cc:725] Time spent opening block manager: real 0.037s user 0.001s sys 0.000s
I20260620 04:27:26.680994 12996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/ts-2/wal
uuid: "0a704308e03e4c2f83672bbb9f35ff79"
format_stamp: "Formatted at 2026-06-20 04:27:26 on dist-test-slave-nk6d"
server_key: "7c55a2a362ce5609a3352f0148609f11"
server_key_iv: "c6fb2e057d342f72b2285fba6e48672c"
server_key_version: "kuduclusterkey@0"
I20260620 04:27:26.681135 12996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:27:26.703227 12996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:27:26.712248 12996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:27:26.712532 12996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:27:26.713315 12996 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:27:26.714426 12996 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:27:26.714526 12996 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:27:26.714627 12996 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:27:26.714694 12996 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:27:26.726459 12996 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35843
I20260620 04:27:26.726553 13124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35843 every 8 connection(s)
I20260620 04:27:26.727717 12996 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/ts-2/data/info.pb
I20260620 04:27:26.734716 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 12996
I20260620 04:27:26.734830 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/ts-2/wal/instance
I20260620 04:27:26.735109 392 external_mini_cluster.cc:1428] Setting key 567f888948e47c23891f052b624ab53b
Jun 20 04:27:26 dist-test-slave-nk6d krb5kdc[12165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929646, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:27:26.744387 12857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:26.730137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37955 (local address 127.0.98.62:46301)
0620 04:27:26.730311 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 04:27:26.730315 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:27:26.731452 (+ 1137us) server_negotiation.cc:407] Connection header received
0620 04:27:26.732718 (+ 1266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:27:26.732722 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:27:26.732781 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:27:26.732883 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:27:26.734329 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:26.735206 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:26.737004 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:26.737352 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:26.737428 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:27:26.740503 (+ 3075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:27:26.740507 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:27:26.740510 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:27:26.740537 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:27:26.742091 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:27:26.742690 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:27:26.742693 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:27:26.742695 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:27:26.742784 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:27:26.743155 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:27:26.743158 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:27:26.743160 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:27:26.743372 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:27:26.743465 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 04:27:26.744094 (+ 629us) server_negotiation.cc:299] Negotiation successful
0620 04:27:26.744219 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 04:27:26.745455 13127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:26.730667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46301 (local address 127.0.98.3:37955)
0620 04:27:26.731228 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:26.731348 (+ 120us) client_negotiation.cc:174] Beginning negotiation
0620 04:27:26.732448 (+ 1100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:27:26.733086 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:27:26.733096 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:27:26.733445 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:27:26.734160 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:27:26.734181 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:27:26.735361 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:27:26.735364 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:27:26.736872 (+ 1508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:27:26.736884 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:27:26.737635 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:27:26.737639 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:27:26.737836 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:27:26.738596 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:27:26.738623 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:27:26.740326 (+ 1703us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:27:26.742208 (+ 1882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:27:26.742215 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:27:26.742233 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:27:26.742578 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:27:26.742870 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:27:26.742873 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:27:26.742875 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:27:26.743051 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:27:26.743470 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:27:26.743477 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:27:26.743811 (+ 334us) client_negotiation.cc:769] Sending connection context
0620 04:27:26.744031 (+ 220us) client_negotiation.cc:240] Negotiation successful
0620 04:27:26.744386 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":137,"threads_started":1}
I20260620 04:27:26.746744 13125 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46301
I20260620 04:27:26.747044 13125 heartbeater.cc:464] Registering TS with master...
I20260620 04:27:26.747656 13125 heartbeater.cc:511] Master 127.0.98.62:46301 requested a full tablet report, sending...
I20260620 04:27:26.748878 12641 ts_manager.cc:194] Registered new tserver with Master: 0a704308e03e4c2f83672bbb9f35ff79 (127.0.98.3:35843)
I20260620 04:27:26.749639 12641 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:37955
I20260620 04:27:26.760710 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 04:27:26.761448 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 12722
I20260620 04:27:26.769361 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 12861
I20260620 04:27:26.775774 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 12996
I20260620 04:27:26.783210 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 12609
I20260620 04:27:26.788502 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46301
--webserver_interface=127.0.98.62
--webserver_port=40161
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46301
--ranger_config_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:44373/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:44373/kms with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:27:26.896901 13128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:27:26.897176 13128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:27:26.897214 13128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:27:26.901271 13128 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:27:26.901356 13128 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:27:26.901376 13128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:27:26.901391 13128 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:27:26.901404 13128 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:27:26.907021 13128 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:44373/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46301
--ranger_config_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46301
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=40161
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13128
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:27:26.908093 13128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:27:26.909111 13128 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:27:26.917238 13133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:27:26.917274 13128 server_base.cc:1034] running on GCE node
W20260620 04:27:26.917285 13136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:27:26.917285 13134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:27:26.918058 13128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:27:26.919076 13128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:27:26.920234 13128 hybrid_clock.cc:648] HybridClock initialized: now 1781929646920208 us; error 36 us; skew 500 ppm
Jun 20 04:27:26 dist-test-slave-nk6d krb5kdc[12165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929646, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:27:26.922914 13128 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 04:27:26.923820 13128 webserver.cc:460] Webserver started at http://127.0.98.62:40161/ using document root <none> and password file <none>
I20260620 04:27:26.924357 13128 fs_manager.cc:359] Metadata directory not provided
I20260620 04:27:26.924423 13128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 04:27:26 dist-test-slave-nk6d krb5kdc[12165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929646, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 04:27:26.970862 13128 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:27:26.973199 13144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:27:26.974551 13128 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 04:27:26.974671 13128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/master-0/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/master-0/wal
uuid: "b8437fed23e6424095fee92cfb17a227"
format_stamp: "Formatted at 2026-06-20 04:27:24 on dist-test-slave-nk6d"
server_key: "60889391fc3fbeadb3a2ddf761024ac5"
server_key_iv: "fc0cbbe599e5ae774a1fa7f493356a1d"
server_key_version: "kuduclusterkey@0"
I20260620 04:27:26.975243 13128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:27:26.993690 13128 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:27:26.996848 13128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:27:26.997028 13128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:27:27.004238 13128 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46301
I20260620 04:27:27.004254 13196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46301 every 8 connection(s)
I20260620 04:27:27.005342 13128 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:27:27.009580 13197 sys_catalog.cc:263] Verifying existing consensus state
I20260620 04:27:27.011212 13197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8437fed23e6424095fee92cfb17a227: Bootstrap starting.
I20260620 04:27:27.015028 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 13128
I20260620 04:27:27.015446 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:44373/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:33171
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:46301
--webserver_port=35253
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:44373/kms with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 04:27:27.027616 13197 log.cc:824] T 00000000000000000000000000000000 P b8437fed23e6424095fee92cfb17a227: Log is configured to *not* fsync() on all Append() calls
I20260620 04:27:27.034435 13197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8437fed23e6424095fee92cfb17a227: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 04:27:27.034806 13197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8437fed23e6424095fee92cfb17a227: Bootstrap complete.
I20260620 04:27:27.039055 13197 raft_consensus.cc:348] T 00000000000000000000000000000000 P b8437fed23e6424095fee92cfb17a227 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8437fed23e6424095fee92cfb17a227" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46301 } }
I20260620 04:27:27.039464 13197 raft_consensus.cc:729] T 00000000000000000000000000000000 P b8437fed23e6424095fee92cfb17a227 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8437fed23e6424095fee92cfb17a227, State: Initialized, Role: FOLLOWER
I20260620 04:27:27.039933 13197 consensus_queue.cc:260] T 00000000000000000000000000000000 P b8437fed23e6424095fee92cfb17a227 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8437fed23e6424095fee92cfb17a227" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46301 } }
I20260620 04:27:27.040091 13197 raft_consensus.cc:388] T 00000000000000000000000000000000 P b8437fed23e6424095fee92cfb17a227 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:27:27.040161 13197 raft_consensus.cc:482] T 00000000000000000000000000000000 P b8437fed23e6424095fee92cfb17a227 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:27:27.040252 13197 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b8437fed23e6424095fee92cfb17a227 [term 1 FOLLOWER]: Advancing to term 2
I20260620 04:27:27.042112 13197 raft_consensus.cc:504] T 00000000000000000000000000000000 P b8437fed23e6424095fee92cfb17a227 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8437fed23e6424095fee92cfb17a227" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46301 } }
I20260620 04:27:27.042423 13197 leader_election.cc:302] T 00000000000000000000000000000000 P b8437fed23e6424095fee92cfb17a227 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b8437fed23e6424095fee92cfb17a227; no voters:
I20260620 04:27:27.042717 13197 leader_election.cc:288] T 00000000000000000000000000000000 P b8437fed23e6424095fee92cfb17a227 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 04:27:27.042847 13201 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b8437fed23e6424095fee92cfb17a227 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 04:27:27.043107 13201 raft_consensus.cc:686] T 00000000000000000000000000000000 P b8437fed23e6424095fee92cfb17a227 [term 2 LEADER]: Becoming Leader. State: Replica: b8437fed23e6424095fee92cfb17a227, State: Running, Role: LEADER
I20260620 04:27:27.043449 13201 consensus_queue.cc:237] T 00000000000000000000000000000000 P b8437fed23e6424095fee92cfb17a227 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8437fed23e6424095fee92cfb17a227" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46301 } }
I20260620 04:27:27.043874 13197 sys_catalog.cc:564] T 00000000000000000000000000000000 P b8437fed23e6424095fee92cfb17a227 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:27:27.044492 13201 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8437fed23e6424095fee92cfb17a227 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8437fed23e6424095fee92cfb17a227. Latest consensus state: current_term: 2 leader_uuid: "b8437fed23e6424095fee92cfb17a227" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8437fed23e6424095fee92cfb17a227" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46301 } } }
I20260620 04:27:27.045116 13202 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8437fed23e6424095fee92cfb17a227 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "b8437fed23e6424095fee92cfb17a227" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8437fed23e6424095fee92cfb17a227" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46301 } } }
I20260620 04:27:27.045253 13201 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8437fed23e6424095fee92cfb17a227 [sys.catalog]: This master's current role is: LEADER
I20260620 04:27:27.045288 13202 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8437fed23e6424095fee92cfb17a227 [sys.catalog]: This master's current role is: LEADER
I20260620 04:27:27.046505 13197 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 04:27:27.049111 13215 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:27:27.053346 13215 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:27:27.054039 13215 catalog_manager.cc:1259] Loaded cluster ID: 55b11160e31b49e9b82ea4776ffa7dab
I20260620 04:27:27.054107 13215 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:27:27.056715 13215 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:27:27.057443 13215 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b8437fed23e6424095fee92cfb17a227: Loaded TSK: 0
I20260620 04:27:27.058728 13215 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 04:27:27.155715 13199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:27:27.156116 13199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:27:27.156195 13199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:27:27.162283 13199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:27:27.162379 13199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:27:27.162467 13199 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 04:27:27.170524 13199 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:44373/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:33171
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=35253
--tserver_master_addrs=127.0.98.62:46301
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13199
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:27:27.172046 13199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:27:27.173426 13199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:27:27.186358 13234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:27:27.186717 13237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:27:27.186790 13235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:27:27.187268 13199 server_base.cc:1034] running on GCE node
I20260620 04:27:27.187767 13199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:27:27.188555 13199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:27:27.189754 13199 hybrid_clock.cc:648] HybridClock initialized: now 1781929647189725 us; error 41 us; skew 500 ppm
Jun 20 04:27:27 dist-test-slave-nk6d krb5kdc[12165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929647, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:27:27.193316 13199 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 04:27:27.194485 13199 webserver.cc:460] Webserver started at http://127.0.98.1:35253/ using document root <none> and password file <none>
I20260620 04:27:27.195178 13199 fs_manager.cc:359] Metadata directory not provided
I20260620 04:27:27.195271 13199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 04:27:27 dist-test-slave-nk6d krb5kdc[12165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929647, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 04:27:27.234441 13199 fs_manager.cc:709] Time spent opening directory manager: real 0.010s user 0.003s sys 0.000s
I20260620 04:27:27.244225 13245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:27:27.246070 13199 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.003s sys 0.000s
I20260620 04:27:27.246217 13199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/ts-0/wal
uuid: "0e23e8c323e54928b36dd233b1ee1d97"
format_stamp: "Formatted at 2026-06-20 04:27:25 on dist-test-slave-nk6d"
server_key: "5811f44c7685eb998833d374ffd839ee"
server_key_iv: "eabb94a43f9afaf3c7b8bfe6317c5929"
server_key_version: "kuduclusterkey@0"
I20260620 04:27:27.246986 13199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:27:27.266980 13199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:27:27.270756 13199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:27:27.270953 13199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:27:27.271651 13199 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:27:27.272601 13199 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:27:27.272651 13199 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:27:27.272708 13199 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:27:27.272740 13199 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:27:27.283638 13199 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33171
I20260620 04:27:27.285089 13199 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/ts-0/data/info.pb
I20260620 04:27:27.286809 13358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33171 every 8 connection(s)
I20260620 04:27:27.293053 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 13199
I20260620 04:27:27.293538 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:44373/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:42783
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:46301
--webserver_port=32779
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:44373/kms with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 04:27:27 dist-test-slave-nk6d krb5kdc[12165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929647, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:27:27.306771 13361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:27.287603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46301 (local address 127.0.98.1:48913)
0620 04:27:27.288103 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:27.288156 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 04:27:27.289142 (+ 986us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:27:27.290555 (+ 1413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:27:27.290564 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:27:27.290906 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:27:27.291533 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:27:27.291553 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:27:27.293820 (+ 2267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:27:27.293825 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:27:27.295934 (+ 2109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:27:27.295945 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:27:27.297254 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:27:27.297258 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:27:27.297444 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:27:27.298358 (+ 914us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:27:27.298379 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:27:27.300156 (+ 1777us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:27:27.302685 (+ 2529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:27:27.302692 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:27:27.302711 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:27:27.303067 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:27:27.303511 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:27:27.303514 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:27:27.303517 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:27:27.303690 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:27:27.304635 (+ 945us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:27:27.304641 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:27:27.304998 (+ 357us) client_negotiation.cc:769] Sending connection context
0620 04:27:27.305224 (+ 226us) client_negotiation.cc:240] Negotiation successful
0620 04:27:27.305598 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":115,"threads_started":1}
I20260620 04:27:27.308676 13362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:27.289079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48913 (local address 127.0.98.62:46301)
0620 04:27:27.289522 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0620 04:27:27.289540 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 04:27:27.289575 (+ 35us) server_negotiation.cc:407] Connection header received
0620 04:27:27.289806 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:27:27.289816 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:27:27.290122 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:27:27.290461 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:27:27.291740 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:27.293445 (+ 1705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:27.296102 (+ 2657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:27.296855 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:27.297012 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:27:27.300402 (+ 3390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:27:27.300409 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:27:27.300430 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:27:27.300465 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:27:27.302530 (+ 2065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:27:27.303231 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:27:27.303238 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:27:27.303244 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:27:27.303402 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:27:27.303800 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:27:27.303803 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:27:27.303805 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:27:27.304365 (+ 560us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:27:27.304572 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0620 04:27:27.307368 (+ 2796us) server_negotiation.cc:299] Negotiation successful
0620 04:27:27.307685 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":116,"threads_started":1}
I20260620 04:27:27.312003 13359 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46301
I20260620 04:27:27.312314 13359 heartbeater.cc:464] Registering TS with master...
I20260620 04:27:27.312856 13359 heartbeater.cc:511] Master 127.0.98.62:46301 requested a full tablet report, sending...
W20260620 04:27:27.315042 13359 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:46301 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 04:27:27.315165 13359 heartbeater.cc:464] Registering TS with master...
I20260620 04:27:27.315335 13359 heartbeater.cc:511] Master 127.0.98.62:46301 requested a full tablet report, sending...
I20260620 04:27:27.316272 13359 heartbeater.cc:464] Registering TS with master...
I20260620 04:27:27.316421 13359 heartbeater.cc:511] Master 127.0.98.62:46301 requested a full tablet report, sending...
W20260620 04:27:27.317137 13359 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 04:27:27.426277 13363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:27:27.426537 13363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:27:27.426595 13363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:27:27.430766 13363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:27:27.430830 13363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:27:27.430909 13363 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 04:27:27.436389 13363 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:44373/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:42783
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=32779
--tserver_master_addrs=127.0.98.62:46301
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13363
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:27:27.437451 13363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:27:27.438520 13363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:27:27.447839 13368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:27:27.448099 13363 server_base.cc:1034] running on GCE node
W20260620 04:27:27.447834 13371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:27:27.448325 13369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:27:27.448752 13363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:27:27.449535 13363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:27:27.450731 13363 hybrid_clock.cc:648] HybridClock initialized: now 1781929647450706 us; error 38 us; skew 500 ppm
Jun 20 04:27:27 dist-test-slave-nk6d krb5kdc[12165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929647, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:27:27.454463 13363 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 04:27:27.455729 13363 webserver.cc:460] Webserver started at http://127.0.98.2:32779/ using document root <none> and password file <none>
I20260620 04:27:27.456589 13363 fs_manager.cc:359] Metadata directory not provided
I20260620 04:27:27.456684 13363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 04:27:27 dist-test-slave-nk6d krb5kdc[12165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929647, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 04:27:27.478574 13363 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 04:27:27.481806 13379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:27:27.483364 13363 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 04:27:27.483487 13363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/ts-1/wal
uuid: "3d6563b6a3a7478e99161bb8178ef39c"
format_stamp: "Formatted at 2026-06-20 04:27:26 on dist-test-slave-nk6d"
server_key: "72c290c7d175efc98c1514d603b2c652"
server_key_iv: "851f4ca665e3af6bd87006ffb31f3f66"
server_key_version: "kuduclusterkey@0"
I20260620 04:27:27.484246 13363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:27:27.512601 13363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:27:27.515642 13363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:27:27.515812 13363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:27:27.516441 13363 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:27:27.517459 13363 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:27:27.517527 13363 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:27:27.517588 13363 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:27:27.517620 13363 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:27:27.528403 13363 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42783
I20260620 04:27:27.528896 13492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42783 every 8 connection(s)
I20260620 04:27:27.529938 13363 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/ts-1/data/info.pb
I20260620 04:27:27.535104 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 13363
I20260620 04:27:27.535614 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:44373/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:35843
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:46301
--webserver_port=42257
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:44373/kms with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 04:27:27 dist-test-slave-nk6d krb5kdc[12165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929647, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:27:27.554908 13495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:27.532916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46301 (local address 127.0.98.2:37051)
0620 04:27:27.534133 (+ 1217us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:27.534193 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 04:27:27.535307 (+ 1114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:27:27.536182 (+ 875us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:27:27.536192 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:27:27.536538 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:27:27.537237 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:27:27.537259 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:27:27.538436 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:27:27.538439 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:27:27.539870 (+ 1431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:27:27.539881 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:27:27.540718 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:27:27.540723 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:27:27.546466 (+ 5743us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:27:27.547240 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:27:27.547264 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:27:27.549466 (+ 2202us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:27:27.551549 (+ 2083us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:27:27.551553 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:27:27.551566 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:27:27.551802 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:27:27.552124 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:27:27.552127 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:27:27.552129 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:27:27.552318 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:27:27.552851 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:27:27.552858 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:27:27.553207 (+ 349us) client_negotiation.cc:769] Sending connection context
0620 04:27:27.553433 (+ 226us) client_negotiation.cc:240] Negotiation successful
0620 04:27:27.553788 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":989,"thread_start_us":139,"threads_started":1}
I20260620 04:27:27.555670 13362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:27.532337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37051 (local address 127.0.98.62:46301)
0620 04:27:27.532489 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 04:27:27.532493 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:27:27.534338 (+ 1845us) server_negotiation.cc:407] Connection header received
0620 04:27:27.535587 (+ 1249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:27:27.535592 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:27:27.535664 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:27:27.535721 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:27:27.537424 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:27.538279 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:27.540026 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:27.540416 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:27.540535 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:27:27.549707 (+ 9172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:27:27.549712 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:27:27.549715 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:27:27.549749 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:27:27.551407 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:27:27.551906 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:27:27.551910 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:27:27.551911 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:27:27.552017 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:27:27.552462 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:27:27.552466 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:27:27.552467 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:27:27.552705 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:27:27.552797 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 04:27:27.555385 (+ 2588us) server_negotiation.cc:299] Negotiation successful
0620 04:27:27.555517 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 04:27:27.556663 13493 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46301
I20260620 04:27:27.556921 13493 heartbeater.cc:464] Registering TS with master...
I20260620 04:27:27.557531 13493 heartbeater.cc:511] Master 127.0.98.62:46301 requested a full tablet report, sending...
W20260620 04:27:27.559193 13493 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:46301 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 04:27:27.559319 13493 heartbeater.cc:464] Registering TS with master...
I20260620 04:27:27.559513 13493 heartbeater.cc:511] Master 127.0.98.62:46301 requested a full tablet report, sending...
I20260620 04:27:27.560348 13493 heartbeater.cc:464] Registering TS with master...
I20260620 04:27:27.560516 13493 heartbeater.cc:511] Master 127.0.98.62:46301 requested a full tablet report, sending...
W20260620 04:27:27.561313 13493 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 04:27:27.671020 13496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:27:27.671314 13496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:27:27.671365 13496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:27:27.675559 13496 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:27:27.675618 13496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:27:27.675730 13496 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 04:27:27.681210 13496 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:44373/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:35843
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=42257
--tserver_master_addrs=127.0.98.62:46301
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13496
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:27:27.682299 13496 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:27:27.683276 13496 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:27:27.692847 13502 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:27:27.692834 13503 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:27:27.692838 13505 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:27:27.694321 13496 server_base.cc:1034] running on GCE node
I20260620 04:27:27.694813 13496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:27:27.695561 13496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:27:27.696777 13496 hybrid_clock.cc:648] HybridClock initialized: now 1781929647696748 us; error 43 us; skew 500 ppm
Jun 20 04:27:27 dist-test-slave-nk6d krb5kdc[12165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929647, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:27:27.699528 13496 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 04:27:27.700511 13496 webserver.cc:460] Webserver started at http://127.0.98.3:42257/ using document root <none> and password file <none>
I20260620 04:27:27.701054 13496 fs_manager.cc:359] Metadata directory not provided
I20260620 04:27:27.701135 13496 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 04:27:27 dist-test-slave-nk6d krb5kdc[12165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929647, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 04:27:27.721742 13496 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 04:27:27.724339 13513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:27:27.725682 13496 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 04:27:27.725796 13496 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/ts-2/wal
uuid: "0a704308e03e4c2f83672bbb9f35ff79"
format_stamp: "Formatted at 2026-06-20 04:27:26 on dist-test-slave-nk6d"
server_key: "7c55a2a362ce5609a3352f0148609f11"
server_key_iv: "c6fb2e057d342f72b2285fba6e48672c"
server_key_version: "kuduclusterkey@0"
I20260620 04:27:27.726349 13496 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:27:27.743343 13496 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:27:27.747258 13496 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:27:27.747466 13496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:27:27.748250 13496 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:27:27.749313 13496 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:27:27.749372 13496 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:27:27.749423 13496 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:27:27.749445 13496 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:27:27.760679 13496 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35843
I20260620 04:27:27.762228 13496 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/ts-2/data/info.pb
I20260620 04:27:27.765429 13626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35843 every 8 connection(s)
I20260620 04:27:27.766173 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 13496
Jun 20 04:27:27 dist-test-slave-nk6d krb5kdc[12165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929647, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:27:27.777731 13631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:27.767116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49984 (local address 127.0.98.62:46301)
0620 04:27:27.767787 (+ 671us) server_negotiation.cc:206] Beginning negotiation
0620 04:27:27.767790 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:27:27.767799 (+ 9us) server_negotiation.cc:407] Connection header received
0620 04:27:27.767835 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:27:27.767837 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:27:27.767869 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:27:27.767937 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:27:27.768676 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:27.769395 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:27.770821 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:27.771183 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:27.771298 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:27:27.774093 (+ 2795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:27:27.774097 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:27:27.774100 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:27:27.774131 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:27:27.776009 (+ 1878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:27:27.776506 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:27:27.776510 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:27:27.776512 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:27:27.776594 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:27:27.776914 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:27:27.776917 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:27:27.776919 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:27:27.777082 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:27:27.777274 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0620 04:27:27.777437 (+ 163us) server_negotiation.cc:299] Negotiation successful
0620 04:27:27.777552 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":614,"spinlock_wait_cycles":2176,"thread_start_us":91,"threads_started":1}
I20260620 04:27:27.779320 13362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:27.766075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36975 (local address 127.0.98.62:46301)
0620 04:27:27.766191 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 04:27:27.766196 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:27:27.766219 (+ 23us) server_negotiation.cc:407] Connection header received
0620 04:27:27.767321 (+ 1102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:27:27.767325 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:27:27.767379 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:27:27.767458 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:27:27.768987 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:27.769636 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:27.771093 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:27.771382 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:27.771500 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:27:27.774341 (+ 2841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:27:27.774345 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:27:27.774346 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:27:27.774400 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:27:27.776138 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:27:27.776752 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:27:27.776755 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:27:27.776757 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:27:27.776825 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:27:27.777213 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:27:27.777216 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:27:27.777218 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:27:27.777435 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:27:27.777500 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 04:27:27.778479 (+ 979us) server_negotiation.cc:299] Negotiation successful
0620 04:27:27.778719 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50,"spinlock_wait_cycles":768}
I20260620 04:27:27.779937 13629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:27.764975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46301 (local address 127.0.98.3:36975)
0620 04:27:27.766033 (+ 1058us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:27.766085 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 04:27:27.767067 (+ 982us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:27:27.767702 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:27:27.767713 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:27:27.768113 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:27:27.768816 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:27:27.768838 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:27:27.769745 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:27:27.769748 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:27:27.770983 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:27:27.770992 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:27:27.771657 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:27:27.771661 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:27:27.771857 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:27:27.772547 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:27:27.772569 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:27:27.774064 (+ 1495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:27:27.776267 (+ 2203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:27:27.776273 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:27:27.776290 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:27:27.776622 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:27:27.776910 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:27:27.776914 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:27:27.776915 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:27:27.777097 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:27:27.777583 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:27:27.777589 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:27:27.778126 (+ 537us) client_negotiation.cc:769] Sending connection context
0620 04:27:27.778438 (+ 312us) client_negotiation.cc:240] Negotiation successful
0620 04:27:27.778789 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":840,"thread_start_us":135,"threads_started":1}
I20260620 04:27:27.781100 13627 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46301
I20260620 04:27:27.781329 13627 heartbeater.cc:464] Registering TS with master...
I20260620 04:27:27.781872 13627 heartbeater.cc:511] Master 127.0.98.62:46301 requested a full tablet report, sending...
W20260620 04:27:27.783360 13627 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:46301 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 04:27:27.783458 13627 heartbeater.cc:464] Registering TS with master...
I20260620 04:27:27.783602 13627 heartbeater.cc:511] Master 127.0.98.62:46301 requested a full tablet report, sending...
I20260620 04:27:27.784449 13627 heartbeater.cc:464] Registering TS with master...
I20260620 04:27:27.784608 13627 heartbeater.cc:511] Master 127.0.98.62:46301 requested a full tablet report, sending...
W20260620 04:27:27.785342 13627 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 20 04:27:28 dist-test-slave-nk6d krb5kdc[12165](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781929648, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:27:28.317358 13359 heartbeater.cc:464] Registering TS with master...
I20260620 04:27:28.317652 13359 heartbeater.cc:511] Master 127.0.98.62:46301 requested a full tablet report, sending...
I20260620 04:27:28.318892 13161 ts_manager.cc:194] Registered new tserver with Master: 0e23e8c323e54928b36dd233b1ee1d97 (127.0.98.1:33171)
I20260620 04:27:28.320626 13161 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:48913
Jun 20 04:27:28 dist-test-slave-nk6d krb5kdc[12165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929648, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 04:27:28.561578 13493 heartbeater.cc:464] Registering TS with master...
I20260620 04:27:28.561909 13493 heartbeater.cc:511] Master 127.0.98.62:46301 requested a full tablet report, sending...
I20260620 04:27:28.562829 13160 ts_manager.cc:194] Registered new tserver with Master: 3d6563b6a3a7478e99161bb8178ef39c (127.0.98.2:42783)
I20260620 04:27:28.563628 13160 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:37051
Jun 20 04:27:28 dist-test-slave-nk6d krb5kdc[12165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929648, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 04:27:28.786394 13627 heartbeater.cc:464] Registering TS with master...
I20260620 04:27:28.786700 13627 heartbeater.cc:511] Master 127.0.98.62:46301 requested a full tablet report, sending...
I20260620 04:27:28.787551 13160 ts_manager.cc:194] Registered new tserver with Master: 0a704308e03e4c2f83672bbb9f35ff79 (127.0.98.3:35843)
I20260620 04:27:28.788379 13160 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:36975
I20260620 04:27:28.792343 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 04:27:28.803831 13649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:28.794041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49996 (local address 127.0.98.62:46301)
0620 04:27:28.794279 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0620 04:27:28.794284 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:27:28.794614 (+ 330us) server_negotiation.cc:407] Connection header received
0620 04:27:28.794726 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:27:28.794730 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:27:28.794801 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:27:28.794895 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:27:28.795677 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:28.796341 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:28.797377 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:28.797723 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:28.797825 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:27:28.799661 (+ 1836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:27:28.799665 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:27:28.799668 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:27:28.799697 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:27:28.801699 (+ 2002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:27:28.802435 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:27:28.802439 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:27:28.802441 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:27:28.802553 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:27:28.802856 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:27:28.802859 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:27:28.802860 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:27:28.803093 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:27:28.803223 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 04:27:28.803499 (+ 276us) server_negotiation.cc:299] Negotiation successful
0620 04:27:28.803629 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":78,"threads_started":1}
I20260620 04:27:28.808573 13160 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49996:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 04:27:28.813227 13160 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 04:27:28.844100 13659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:28.828297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35843 (local address 127.0.0.1:45612)
0620 04:27:28.833686 (+ 5389us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:28.833753 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0620 04:27:28.833897 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:27:28.835555 (+ 1658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:27:28.835558 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:27:28.835574 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:27:28.835658 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:27:28.835664 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:27:28.838803 (+ 3139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:27:28.838806 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:27:28.841995 (+ 3189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:27:28.842007 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:27:28.843412 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:27:28.843415 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:27:28.843544 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:27:28.843584 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 04:27:28.843740 (+ 156us) client_negotiation.cc:240] Negotiation successful
0620 04:27:28.843911 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5260,"mutex_wait_us":3,"thread_start_us":87,"threads_started":1}
I20260620 04:27:28.847373 13657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:28.827631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50870 (local address 127.0.98.2:42783)
0620 04:27:28.828158 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0620 04:27:28.828165 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 04:27:28.833861 (+ 5696us) server_negotiation.cc:407] Connection header received
0620 04:27:28.834497 (+ 636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:27:28.834504 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:27:28.834744 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:27:28.834897 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:27:28.835940 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:28.837105 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:28.844904 (+ 7799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:28.845880 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:28.845966 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:27:28.846130 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 04:27:28.847083 (+ 953us) server_negotiation.cc:299] Negotiation successful
0620 04:27:28.847208 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":89,"threads_started":1}
I20260620 04:27:28.847806 13662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:28.828974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45612 (local address 127.0.98.3:35843)
0620 04:27:28.829200 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 04:27:28.829207 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 04:27:28.834040 (+ 4833us) server_negotiation.cc:407] Connection header received
0620 04:27:28.834126 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:27:28.834132 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:27:28.835266 (+ 1134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:27:28.835407 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:27:28.835769 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:28.838685 (+ 2916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:28.842138 (+ 3453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:28.843074 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:28.843181 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:27:28.843336 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 04:27:28.846092 (+ 2756us) server_negotiation.cc:299] Negotiation successful
0620 04:27:28.847659 (+ 1567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":66,"threads_started":1}
I20260620 04:27:28.849310 13660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:28.827498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42783 (local address 127.0.0.1:50870)
0620 04:27:28.833688 (+ 6190us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:28.833753 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0620 04:27:28.834320 (+ 567us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:27:28.835135 (+ 815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:27:28.835141 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:27:28.835166 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:27:28.835307 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:27:28.835315 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:27:28.837751 (+ 2436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:27:28.837754 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:27:28.839200 (+ 1446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:27:28.839212 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:27:28.846779 (+ 7567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:27:28.846783 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:27:28.846903 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:27:28.846929 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 04:27:28.848763 (+ 1834us) client_negotiation.cc:240] Negotiation successful
0620 04:27:28.848880 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5977,"thread_start_us":99,"threads_started":1}
I20260620 04:27:28.852486 13561 tablet_service.cc:1467] Processing CreateTablet for tablet b38582ef77a1406e90fce1fae3f6b522 (DEFAULT_TABLE table=test-table [id=3368bfb88c1b43d8b7e1caa000541464]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:27:28.853765 13561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b38582ef77a1406e90fce1fae3f6b522. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:27:28.858363 13427 tablet_service.cc:1467] Processing CreateTablet for tablet b38582ef77a1406e90fce1fae3f6b522 (DEFAULT_TABLE table=test-table [id=3368bfb88c1b43d8b7e1caa000541464]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:27:28.859587 13427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b38582ef77a1406e90fce1fae3f6b522. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:27:28.879231 13656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:28.828591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33171 (local address 127.0.0.1:47362)
0620 04:27:28.840759 (+ 12168us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:28.840775 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 04:27:28.840883 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:27:28.844622 (+ 3739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:27:28.844626 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:27:28.844648 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:27:28.844741 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:27:28.844747 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:27:28.868181 (+ 23434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:27:28.868185 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:27:28.869458 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:27:28.869474 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:27:28.878714 (+ 9240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:27:28.878718 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:27:28.878880 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:27:28.878914 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 04:27:28.878974 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 04:27:28.879065 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12083,"thread_start_us":79,"threads_started":1}
I20260620 04:27:28.880121 13661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:28.828772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47362 (local address 127.0.98.1:33171)
0620 04:27:28.840886 (+ 12114us) server_negotiation.cc:206] Beginning negotiation
0620 04:27:28.840894 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 04:27:28.840924 (+ 30us) server_negotiation.cc:407] Connection header received
0620 04:27:28.841009 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:27:28.841015 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:27:28.841253 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:27:28.841420 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:27:28.845288 (+ 3868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:28.866534 (+ 21246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:28.877532 (+ 10998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:28.878571 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:28.879601 (+ 1030us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:27:28.879785 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0620 04:27:28.879855 (+ 70us) server_negotiation.cc:299] Negotiation successful
0620 04:27:28.879979 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11988,"thread_start_us":121,"threads_started":1}
I20260620 04:27:28.883097 13293 tablet_service.cc:1467] Processing CreateTablet for tablet b38582ef77a1406e90fce1fae3f6b522 (DEFAULT_TABLE table=test-table [id=3368bfb88c1b43d8b7e1caa000541464]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:27:28.884078 13293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b38582ef77a1406e90fce1fae3f6b522. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:27:28.902169 13663 tablet_bootstrap.cc:492] T b38582ef77a1406e90fce1fae3f6b522 P 0a704308e03e4c2f83672bbb9f35ff79: Bootstrap starting.
I20260620 04:27:28.904351 13664 tablet_bootstrap.cc:492] T b38582ef77a1406e90fce1fae3f6b522 P 0e23e8c323e54928b36dd233b1ee1d97: Bootstrap starting.
I20260620 04:27:28.904443 13665 tablet_bootstrap.cc:492] T b38582ef77a1406e90fce1fae3f6b522 P 3d6563b6a3a7478e99161bb8178ef39c: Bootstrap starting.
I20260620 04:27:28.904868 13663 tablet_bootstrap.cc:654] T b38582ef77a1406e90fce1fae3f6b522 P 0a704308e03e4c2f83672bbb9f35ff79: Neither blocks nor log segments found. Creating new log.
I20260620 04:27:28.905822 13663 log.cc:824] T b38582ef77a1406e90fce1fae3f6b522 P 0a704308e03e4c2f83672bbb9f35ff79: Log is configured to *not* fsync() on all Append() calls
I20260620 04:27:28.906661 13664 tablet_bootstrap.cc:654] T b38582ef77a1406e90fce1fae3f6b522 P 0e23e8c323e54928b36dd233b1ee1d97: Neither blocks nor log segments found. Creating new log.
I20260620 04:27:28.908169 13663 tablet_bootstrap.cc:492] T b38582ef77a1406e90fce1fae3f6b522 P 0a704308e03e4c2f83672bbb9f35ff79: No bootstrap required, opened a new log
I20260620 04:27:28.908358 13663 ts_tablet_manager.cc:1397] T b38582ef77a1406e90fce1fae3f6b522 P 0a704308e03e4c2f83672bbb9f35ff79: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260620 04:27:28.912843 13663 raft_consensus.cc:348] T b38582ef77a1406e90fce1fae3f6b522 P 0a704308e03e4c2f83672bbb9f35ff79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d6563b6a3a7478e99161bb8178ef39c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42783 } } peers { permanent_uuid: "0a704308e03e4c2f83672bbb9f35ff79" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35843 } } peers { permanent_uuid: "0e23e8c323e54928b36dd233b1ee1d97" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33171 } }
I20260620 04:27:28.913178 13663 raft_consensus.cc:374] T b38582ef77a1406e90fce1fae3f6b522 P 0a704308e03e4c2f83672bbb9f35ff79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:27:28.913224 13663 raft_consensus.cc:729] T b38582ef77a1406e90fce1fae3f6b522 P 0a704308e03e4c2f83672bbb9f35ff79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a704308e03e4c2f83672bbb9f35ff79, State: Initialized, Role: FOLLOWER
I20260620 04:27:28.913960 13663 consensus_queue.cc:260] T b38582ef77a1406e90fce1fae3f6b522 P 0a704308e03e4c2f83672bbb9f35ff79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d6563b6a3a7478e99161bb8178ef39c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42783 } } peers { permanent_uuid: "0a704308e03e4c2f83672bbb9f35ff79" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35843 } } peers { permanent_uuid: "0e23e8c323e54928b36dd233b1ee1d97" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33171 } }
I20260620 04:27:28.915225 13663 ts_tablet_manager.cc:1428] T b38582ef77a1406e90fce1fae3f6b522 P 0a704308e03e4c2f83672bbb9f35ff79: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260620 04:27:28.915629 13627 heartbeater.cc:503] Master 127.0.98.62:46301 was elected leader, sending a full tablet report...
I20260620 04:27:28.916917 13664 log.cc:824] T b38582ef77a1406e90fce1fae3f6b522 P 0e23e8c323e54928b36dd233b1ee1d97: Log is configured to *not* fsync() on all Append() calls
I20260620 04:27:28.917408 13665 tablet_bootstrap.cc:654] T b38582ef77a1406e90fce1fae3f6b522 P 3d6563b6a3a7478e99161bb8178ef39c: Neither blocks nor log segments found. Creating new log.
I20260620 04:27:28.918336 13665 log.cc:824] T b38582ef77a1406e90fce1fae3f6b522 P 3d6563b6a3a7478e99161bb8178ef39c: Log is configured to *not* fsync() on all Append() calls
I20260620 04:27:28.921908 13664 tablet_bootstrap.cc:492] T b38582ef77a1406e90fce1fae3f6b522 P 0e23e8c323e54928b36dd233b1ee1d97: No bootstrap required, opened a new log
I20260620 04:27:28.922122 13664 ts_tablet_manager.cc:1397] T b38582ef77a1406e90fce1fae3f6b522 P 0e23e8c323e54928b36dd233b1ee1d97: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.000s
I20260620 04:27:28.924641 13665 tablet_bootstrap.cc:492] T b38582ef77a1406e90fce1fae3f6b522 P 3d6563b6a3a7478e99161bb8178ef39c: No bootstrap required, opened a new log
I20260620 04:27:28.924860 13665 ts_tablet_manager.cc:1397] T b38582ef77a1406e90fce1fae3f6b522 P 3d6563b6a3a7478e99161bb8178ef39c: Time spent bootstrapping tablet: real 0.021s user 0.005s sys 0.000s
I20260620 04:27:28.926410 13664 raft_consensus.cc:348] T b38582ef77a1406e90fce1fae3f6b522 P 0e23e8c323e54928b36dd233b1ee1d97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d6563b6a3a7478e99161bb8178ef39c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42783 } } peers { permanent_uuid: "0a704308e03e4c2f83672bbb9f35ff79" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35843 } } peers { permanent_uuid: "0e23e8c323e54928b36dd233b1ee1d97" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33171 } }
I20260620 04:27:28.926713 13664 raft_consensus.cc:374] T b38582ef77a1406e90fce1fae3f6b522 P 0e23e8c323e54928b36dd233b1ee1d97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:27:28.926755 13664 raft_consensus.cc:729] T b38582ef77a1406e90fce1fae3f6b522 P 0e23e8c323e54928b36dd233b1ee1d97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e23e8c323e54928b36dd233b1ee1d97, State: Initialized, Role: FOLLOWER
I20260620 04:27:28.927459 13664 consensus_queue.cc:260] T b38582ef77a1406e90fce1fae3f6b522 P 0e23e8c323e54928b36dd233b1ee1d97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d6563b6a3a7478e99161bb8178ef39c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42783 } } peers { permanent_uuid: "0a704308e03e4c2f83672bbb9f35ff79" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35843 } } peers { permanent_uuid: "0e23e8c323e54928b36dd233b1ee1d97" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33171 } }
I20260620 04:27:28.928685 13664 ts_tablet_manager.cc:1428] T b38582ef77a1406e90fce1fae3f6b522 P 0e23e8c323e54928b36dd233b1ee1d97: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260620 04:27:28.929093 13359 heartbeater.cc:503] Master 127.0.98.62:46301 was elected leader, sending a full tablet report...
I20260620 04:27:28.940017 13665 raft_consensus.cc:348] T b38582ef77a1406e90fce1fae3f6b522 P 3d6563b6a3a7478e99161bb8178ef39c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d6563b6a3a7478e99161bb8178ef39c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42783 } } peers { permanent_uuid: "0a704308e03e4c2f83672bbb9f35ff79" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35843 } } peers { permanent_uuid: "0e23e8c323e54928b36dd233b1ee1d97" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33171 } }
I20260620 04:27:28.940518 13665 raft_consensus.cc:374] T b38582ef77a1406e90fce1fae3f6b522 P 3d6563b6a3a7478e99161bb8178ef39c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:27:28.940639 13665 raft_consensus.cc:729] T b38582ef77a1406e90fce1fae3f6b522 P 3d6563b6a3a7478e99161bb8178ef39c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d6563b6a3a7478e99161bb8178ef39c, State: Initialized, Role: FOLLOWER
I20260620 04:27:28.941288 13665 consensus_queue.cc:260] T b38582ef77a1406e90fce1fae3f6b522 P 3d6563b6a3a7478e99161bb8178ef39c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d6563b6a3a7478e99161bb8178ef39c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42783 } } peers { permanent_uuid: "0a704308e03e4c2f83672bbb9f35ff79" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35843 } } peers { permanent_uuid: "0e23e8c323e54928b36dd233b1ee1d97" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33171 } }
I20260620 04:27:28.942610 13665 ts_tablet_manager.cc:1428] T b38582ef77a1406e90fce1fae3f6b522 P 3d6563b6a3a7478e99161bb8178ef39c: Time spent starting tablet: real 0.018s user 0.003s sys 0.004s
I20260620 04:27:28.943028 13493 heartbeater.cc:503] Master 127.0.98.62:46301 was elected leader, sending a full tablet report...
W20260620 04:27:29.016780 13628 tablet.cc:2369] T b38582ef77a1406e90fce1fae3f6b522 P 0a704308e03e4c2f83672bbb9f35ff79: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 04:27:29.030761 13671 raft_consensus.cc:482] T b38582ef77a1406e90fce1fae3f6b522 P 3d6563b6a3a7478e99161bb8178ef39c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:27:29.031003 13671 raft_consensus.cc:504] T b38582ef77a1406e90fce1fae3f6b522 P 3d6563b6a3a7478e99161bb8178ef39c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d6563b6a3a7478e99161bb8178ef39c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42783 } } peers { permanent_uuid: "0a704308e03e4c2f83672bbb9f35ff79" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35843 } } peers { permanent_uuid: "0e23e8c323e54928b36dd233b1ee1d97" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33171 } }
I20260620 04:27:29.032483 13671 leader_election.cc:288] T b38582ef77a1406e90fce1fae3f6b522 P 3d6563b6a3a7478e99161bb8178ef39c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a704308e03e4c2f83672bbb9f35ff79 (127.0.98.3:35843), 0e23e8c323e54928b36dd233b1ee1d97 (127.0.98.1:33171)
W20260620 04:27:29.033429 13494 tablet.cc:2369] T b38582ef77a1406e90fce1fae3f6b522 P 3d6563b6a3a7478e99161bb8178ef39c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 04:27:29.038647 13360 tablet.cc:2369] T b38582ef77a1406e90fce1fae3f6b522 P 0e23e8c323e54928b36dd233b1ee1d97: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 04:27:29.039121 13662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:29.033944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41425 (local address 127.0.98.3:35843)
0620 04:27:29.034058 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 04:27:29.034061 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:27:29.034333 (+ 272us) server_negotiation.cc:407] Connection header received
0620 04:27:29.034481 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:27:29.034485 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:27:29.034549 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:27:29.034647 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:27:29.035113 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:29.035902 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:29.037707 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:29.038335 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:29.038424 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:27:29.038496 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 04:27:29.038881 (+ 385us) server_negotiation.cc:299] Negotiation successful
0620 04:27:29.038972 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 04:27:29.039544 13673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:29.032968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35843 (local address 127.0.98.2:41425)
0620 04:27:29.034204 (+ 1236us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:29.034218 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:27:29.034339 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:27:29.034777 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:27:29.034781 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:27:29.034803 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:27:29.034964 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:27:29.034973 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:27:29.036118 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:27:29.036122 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:27:29.037534 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:27:29.037547 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:27:29.038598 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:27:29.038602 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:27:29.038719 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:27:29.038747 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 04:27:29.039268 (+ 521us) client_negotiation.cc:240] Negotiation successful
0620 04:27:29.039368 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1172,"thread_start_us":109,"threads_started":1}
I20260620 04:27:29.040213 13661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:29.033697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49501 (local address 127.0.98.1:33171)
0620 04:27:29.033819 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 04:27:29.033823 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:27:29.034684 (+ 861us) server_negotiation.cc:407] Connection header received
0620 04:27:29.034855 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:27:29.034858 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:27:29.034920 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:27:29.034965 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:27:29.035334 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:29.036150 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:29.037668 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:29.038313 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:29.038393 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:27:29.038463 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 04:27:29.040028 (+ 1565us) server_negotiation.cc:299] Negotiation successful
0620 04:27:29.040107 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 04:27:29.040313 13581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b38582ef77a1406e90fce1fae3f6b522" candidate_uuid: "3d6563b6a3a7478e99161bb8178ef39c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a704308e03e4c2f83672bbb9f35ff79" is_pre_election: true
I20260620 04:27:29.040614 13672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:29.033331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33171 (local address 127.0.98.2:49501)
0620 04:27:29.034607 (+ 1276us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:29.034619 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 04:27:29.034747 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:27:29.035091 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:27:29.035093 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:27:29.035108 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:27:29.035223 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:27:29.035230 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:27:29.036284 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:27:29.036287 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:27:29.037522 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:27:29.037533 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:27:29.039530 (+ 1997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:27:29.039533 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:27:29.039859 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:27:29.039883 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 04:27:29.040399 (+ 516us) client_negotiation.cc:240] Negotiation successful
0620 04:27:29.040474 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1215,"mutex_wait_us":1,"thread_start_us":96,"threads_started":1}
I20260620 04:27:29.040655 13581 raft_consensus.cc:2454] T b38582ef77a1406e90fce1fae3f6b522 P 0a704308e03e4c2f83672bbb9f35ff79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d6563b6a3a7478e99161bb8178ef39c in term 0.
I20260620 04:27:29.041145 13383 leader_election.cc:302] T b38582ef77a1406e90fce1fae3f6b522 P 3d6563b6a3a7478e99161bb8178ef39c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a704308e03e4c2f83672bbb9f35ff79, 3d6563b6a3a7478e99161bb8178ef39c; no voters:
I20260620 04:27:29.041234 13305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b38582ef77a1406e90fce1fae3f6b522" candidate_uuid: "3d6563b6a3a7478e99161bb8178ef39c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e23e8c323e54928b36dd233b1ee1d97" is_pre_election: true
I20260620 04:27:29.041505 13305 raft_consensus.cc:2454] T b38582ef77a1406e90fce1fae3f6b522 P 0e23e8c323e54928b36dd233b1ee1d97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d6563b6a3a7478e99161bb8178ef39c in term 0.
I20260620 04:27:29.041805 13671 raft_consensus.cc:2781] T b38582ef77a1406e90fce1fae3f6b522 P 3d6563b6a3a7478e99161bb8178ef39c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 04:27:29.041893 13671 raft_consensus.cc:482] T b38582ef77a1406e90fce1fae3f6b522 P 3d6563b6a3a7478e99161bb8178ef39c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 04:27:29.041942 13671 raft_consensus.cc:3037] T b38582ef77a1406e90fce1fae3f6b522 P 3d6563b6a3a7478e99161bb8178ef39c [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:27:29.043212 13671 raft_consensus.cc:504] T b38582ef77a1406e90fce1fae3f6b522 P 3d6563b6a3a7478e99161bb8178ef39c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d6563b6a3a7478e99161bb8178ef39c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42783 } } peers { permanent_uuid: "0a704308e03e4c2f83672bbb9f35ff79" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35843 } } peers { permanent_uuid: "0e23e8c323e54928b36dd233b1ee1d97" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33171 } }
I20260620 04:27:29.043586 13671 leader_election.cc:288] T b38582ef77a1406e90fce1fae3f6b522 P 3d6563b6a3a7478e99161bb8178ef39c [CANDIDATE]: Term 1 election: Requested vote from peers 0a704308e03e4c2f83672bbb9f35ff79 (127.0.98.3:35843), 0e23e8c323e54928b36dd233b1ee1d97 (127.0.98.1:33171)
I20260620 04:27:29.044082 13305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b38582ef77a1406e90fce1fae3f6b522" candidate_uuid: "3d6563b6a3a7478e99161bb8178ef39c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e23e8c323e54928b36dd233b1ee1d97"
I20260620 04:27:29.044130 13581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b38582ef77a1406e90fce1fae3f6b522" candidate_uuid: "3d6563b6a3a7478e99161bb8178ef39c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a704308e03e4c2f83672bbb9f35ff79"
I20260620 04:27:29.044199 13305 raft_consensus.cc:3037] T b38582ef77a1406e90fce1fae3f6b522 P 0e23e8c323e54928b36dd233b1ee1d97 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:27:29.044272 13581 raft_consensus.cc:3037] T b38582ef77a1406e90fce1fae3f6b522 P 0a704308e03e4c2f83672bbb9f35ff79 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:27:29.045379 13305 raft_consensus.cc:2454] T b38582ef77a1406e90fce1fae3f6b522 P 0e23e8c323e54928b36dd233b1ee1d97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d6563b6a3a7478e99161bb8178ef39c in term 1.
I20260620 04:27:29.045688 13383 leader_election.cc:302] T b38582ef77a1406e90fce1fae3f6b522 P 3d6563b6a3a7478e99161bb8178ef39c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e23e8c323e54928b36dd233b1ee1d97, 3d6563b6a3a7478e99161bb8178ef39c; no voters:
I20260620 04:27:29.045708 13581 raft_consensus.cc:2454] T b38582ef77a1406e90fce1fae3f6b522 P 0a704308e03e4c2f83672bbb9f35ff79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d6563b6a3a7478e99161bb8178ef39c in term 1.
I20260620 04:27:29.045855 13671 raft_consensus.cc:2781] T b38582ef77a1406e90fce1fae3f6b522 P 3d6563b6a3a7478e99161bb8178ef39c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:27:29.046068 13671 raft_consensus.cc:686] T b38582ef77a1406e90fce1fae3f6b522 P 3d6563b6a3a7478e99161bb8178ef39c [term 1 LEADER]: Becoming Leader. State: Replica: 3d6563b6a3a7478e99161bb8178ef39c, State: Running, Role: LEADER
I20260620 04:27:29.046399 13671 consensus_queue.cc:237] T b38582ef77a1406e90fce1fae3f6b522 P 3d6563b6a3a7478e99161bb8178ef39c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d6563b6a3a7478e99161bb8178ef39c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42783 } } peers { permanent_uuid: "0a704308e03e4c2f83672bbb9f35ff79" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35843 } } peers { permanent_uuid: "0e23e8c323e54928b36dd233b1ee1d97" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33171 } }
I20260620 04:27:29.049969 13160 catalog_manager.cc:5495] T b38582ef77a1406e90fce1fae3f6b522 P 3d6563b6a3a7478e99161bb8178ef39c reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d6563b6a3a7478e99161bb8178ef39c (127.0.98.2). New cstate: current_term: 1 leader_uuid: "3d6563b6a3a7478e99161bb8178ef39c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d6563b6a3a7478e99161bb8178ef39c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a704308e03e4c2f83672bbb9f35ff79" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e23e8c323e54928b36dd233b1ee1d97" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33171 } health_report { overall_health: UNKNOWN } } }
I20260620 04:27:29.073144 13657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:29.067789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50872 (local address 127.0.98.2:42783)
0620 04:27:29.068113 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0620 04:27:29.068119 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 04:27:29.068138 (+ 19us) server_negotiation.cc:407] Connection header received
0620 04:27:29.068236 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:27:29.068240 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:27:29.068321 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:27:29.068458 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:27:29.068783 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:29.069529 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:29.070795 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:29.071137 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:29.071236 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:27:29.071698 (+ 462us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:27:29.072319 (+ 621us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:27:29.072444 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 04:27:29.072885 (+ 441us) server_negotiation.cc:299] Negotiation successful
0620 04:27:29.072955 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228}
I20260620 04:27:29.080772 13305 raft_consensus.cc:1261] T b38582ef77a1406e90fce1fae3f6b522 P 0e23e8c323e54928b36dd233b1ee1d97 [term 1 FOLLOWER]: Refusing update from remote peer 3d6563b6a3a7478e99161bb8178ef39c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:27:29.080772 13581 raft_consensus.cc:1261] T b38582ef77a1406e90fce1fae3f6b522 P 0a704308e03e4c2f83672bbb9f35ff79 [term 1 FOLLOWER]: Refusing update from remote peer 3d6563b6a3a7478e99161bb8178ef39c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:27:29.081626 13671 consensus_queue.cc:1035] T b38582ef77a1406e90fce1fae3f6b522 P 3d6563b6a3a7478e99161bb8178ef39c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a704308e03e4c2f83672bbb9f35ff79" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:27:29.081854 13674 consensus_queue.cc:1035] T b38582ef77a1406e90fce1fae3f6b522 P 3d6563b6a3a7478e99161bb8178ef39c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e23e8c323e54928b36dd233b1ee1d97" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:27:29.091982 13685 mvcc.cc:205] Tried to move back new op lower bound from 7298783842624630784 to 7298783842496573440. Current Snapshot: MvccSnapshot[applied={T|T < 7298783842624630784}]
I20260620 04:27:29.092981 13686 mvcc.cc:205] Tried to move back new op lower bound from 7298783842624630784 to 7298783842496573440. Current Snapshot: MvccSnapshot[applied={T|T < 7298783842624630784}]
I20260620 04:27:29.101109 13161 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49996:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:27:29.101279 13161 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49996:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:27:29.103684 13161 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b8437fed23e6424095fee92cfb17a227: Sending DeleteTablet for 3 replicas of tablet b38582ef77a1406e90fce1fae3f6b522
I20260620 04:27:29.104566 13427 tablet_service.cc:1514] Processing DeleteTablet for tablet b38582ef77a1406e90fce1fae3f6b522 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:27:29 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50870
I20260620 04:27:29.104668 13293 tablet_service.cc:1514] Processing DeleteTablet for tablet b38582ef77a1406e90fce1fae3f6b522 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:27:29 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47362
I20260620 04:27:29.105053 13693 tablet_replica.cc:330] T b38582ef77a1406e90fce1fae3f6b522 P 3d6563b6a3a7478e99161bb8178ef39c: stopping tablet replica
I20260620 04:27:29.105053 13561 tablet_service.cc:1514] Processing DeleteTablet for tablet b38582ef77a1406e90fce1fae3f6b522 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:27:29 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45612
I20260620 04:27:29.105379 13694 tablet_replica.cc:330] T b38582ef77a1406e90fce1fae3f6b522 P 0e23e8c323e54928b36dd233b1ee1d97: stopping tablet replica
I20260620 04:27:29.105419 13693 raft_consensus.cc:2229] T b38582ef77a1406e90fce1fae3f6b522 P 3d6563b6a3a7478e99161bb8178ef39c [term 1 LEADER]: Raft consensus shutting down.
I20260620 04:27:29.105600 13695 tablet_replica.cc:330] T b38582ef77a1406e90fce1fae3f6b522 P 0a704308e03e4c2f83672bbb9f35ff79: stopping tablet replica
I20260620 04:27:29.105726 13694 raft_consensus.cc:2229] T b38582ef77a1406e90fce1fae3f6b522 P 0e23e8c323e54928b36dd233b1ee1d97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:27:29.105756 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 13199
I20260620 04:27:29.105865 13693 raft_consensus.cc:2258] T b38582ef77a1406e90fce1fae3f6b522 P 3d6563b6a3a7478e99161bb8178ef39c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:27:29.105942 13695 raft_consensus.cc:2229] T b38582ef77a1406e90fce1fae3f6b522 P 0a704308e03e4c2f83672bbb9f35ff79 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:27:29.106081 13694 raft_consensus.cc:2258] T b38582ef77a1406e90fce1fae3f6b522 P 0e23e8c323e54928b36dd233b1ee1d97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:27:29.106400 13695 raft_consensus.cc:2258] T b38582ef77a1406e90fce1fae3f6b522 P 0a704308e03e4c2f83672bbb9f35ff79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:27:29.108044 13695 ts_tablet_manager.cc:1905] T b38582ef77a1406e90fce1fae3f6b522 P 0a704308e03e4c2f83672bbb9f35ff79: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:27:29.108573 13693 ts_tablet_manager.cc:1905] T b38582ef77a1406e90fce1fae3f6b522 P 3d6563b6a3a7478e99161bb8178ef39c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:27:29.110761 13695 ts_tablet_manager.cc:1918] T b38582ef77a1406e90fce1fae3f6b522 P 0a704308e03e4c2f83672bbb9f35ff79: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:27:29.110929 13695 log.cc:1196] T b38582ef77a1406e90fce1fae3f6b522 P 0a704308e03e4c2f83672bbb9f35ff79: Deleting WAL directory at /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/ts-2/wal/wals/b38582ef77a1406e90fce1fae3f6b522
I20260620 04:27:29.111038 13693 ts_tablet_manager.cc:1918] T b38582ef77a1406e90fce1fae3f6b522 P 3d6563b6a3a7478e99161bb8178ef39c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:27:29.111147 13693 log.cc:1196] T b38582ef77a1406e90fce1fae3f6b522 P 3d6563b6a3a7478e99161bb8178ef39c: Deleting WAL directory at /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929496330398-392-0/minicluster-data/ts-1/wal/wals/b38582ef77a1406e90fce1fae3f6b522
I20260620 04:27:29.111390 13695 ts_tablet_manager.cc:1939] T b38582ef77a1406e90fce1fae3f6b522 P 0a704308e03e4c2f83672bbb9f35ff79: Deleting consensus metadata
I20260620 04:27:29.111433 13693 ts_tablet_manager.cc:1939] T b38582ef77a1406e90fce1fae3f6b522 P 3d6563b6a3a7478e99161bb8178ef39c: Deleting consensus metadata
I20260620 04:27:29.112644 13148 catalog_manager.cc:4841] TS 0a704308e03e4c2f83672bbb9f35ff79 (127.0.98.3:35843): tablet b38582ef77a1406e90fce1fae3f6b522 (table test-table [id=3368bfb88c1b43d8b7e1caa000541464]) successfully deleted
I20260620 04:27:29.112846 13148 catalog_manager.cc:4841] TS 3d6563b6a3a7478e99161bb8178ef39c (127.0.98.2:42783): tablet b38582ef77a1406e90fce1fae3f6b522 (table test-table [id=3368bfb88c1b43d8b7e1caa000541464]) successfully deleted
W20260620 04:27:29.113580 13148 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33171 (error 108)
I20260620 04:27:29.114012 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 13363
I20260620 04:27:29.115000 13656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:29.114693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33171 (local address 127.0.0.1:47366)
0620 04:27:29.114835 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:29.114898 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260620 04:27:29.115142 13148 catalog_manager.cc:4584] TS 0e23e8c323e54928b36dd233b1ee1d97 (127.0.98.1:33171): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b38582ef77a1406e90fce1fae3f6b522: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33171: connect: Connection refused (error 111)
I20260620 04:27:29.119194 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 13496
I20260620 04:27:29.123927 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 13128
I20260620 04:27:29.131381 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 04:27:29.466564 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 04:27:29.466665 392 mini_ranger.cc:66] Stopping Ranger...
I20260620 04:27:29.801440 392 mini_ranger.cc:68] Stopped Ranger
2026-06-20 04:27:29.801 UTC [12195] LOG: received smart shutdown request
2026-06-20 04:27:29.805 UTC [12195] LOG: background worker "logical replication launcher" (PID 12204) exited with exit code 1
2026-06-20 04:27:29.805 UTC [12199] LOG: shutting down
2026-06-20 04:27:29.947 UTC [12195] LOG: database system is shut down
2026-06-20T04:27:29Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (46634 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929496330398-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:27:30 dist-test-slave-nk6d krb5kdc[13705](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:27:30 dist-test-slave-nk6d krb5kdc[13705](info): set up 2 sockets
Jun 20 04:27:30 dist-test-slave-nk6d krb5kdc[13705](info): commencing operation
krb5kdc: starting...
W20260620 04:27:30.983780 392 mini_kdc.cc:121] Time spent starting KDC: real 0.923s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929496330398-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929496330398-392-0/krb5kdc/test-user.keytab.
Jun 20 04:27:31 dist-test-slave-nk6d krb5kdc[13705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929651, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:27:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:27:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 04:27:31.099749 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929496330398-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929496330398-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929496330398-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34529
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34529,127.0.98.61:37441,127.0.98.60:37985
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:27:31.207341 13718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:27:31.207628 13718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:27:31.207669 13718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:27:31.211782 13718 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:27:31.211855 13718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:27:31.211874 13718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:27:31.211890 13718 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:27:31.211903 13718 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:27:31.217218 13718 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929496330398-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929496330398-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34529,127.0.98.61:37441,127.0.98.60:37985
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34529
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13718
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929496330398-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:27:31.218288 13718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:27:31.219316 13718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:27:31.227533 13724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:27:31.227551 13723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:27:31.227756 13718 server_base.cc:1034] running on GCE node
W20260620 04:27:31.227551 13726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:27:31.228499 13718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:27:31.229606 13718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:27:31.230757 13718 hybrid_clock.cc:648] HybridClock initialized: now 1781929651230732 us; error 39 us; skew 500 ppm
Jun 20 04:27:31 dist-test-slave-nk6d krb5kdc[13705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929651, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:27:31.233417 13718 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 04:27:31.234300 13718 webserver.cc:460] Webserver started at http://127.0.98.62:39107/ using document root <none> and password file <none>
I20260620 04:27:31.234825 13718 fs_manager.cc:359] Metadata directory not provided
I20260620 04:27:31.234905 13718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:27:31.235080 13718 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:27:31.237074 13718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929496330398-392-0/minicluster-data/master-0/data/instance:
uuid: "31cb951a80fc476f97c0a9298b27f27e"
format_stamp: "Formatted at 2026-06-20 04:27:31 on dist-test-slave-nk6d"
server_key: "e8cf6c9e9babbc3aa373248a070fd25a"
server_key_iv: "af747955f5858487f59c9b710b459615"
server_key_version: "encryptionkey@0"
I20260620 04:27:31.237577 13718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929496330398-392-0/minicluster-data/master-0/wal/instance:
uuid: "31cb951a80fc476f97c0a9298b27f27e"
format_stamp: "Formatted at 2026-06-20 04:27:31 on dist-test-slave-nk6d"
server_key: "e8cf6c9e9babbc3aa373248a070fd25a"
server_key_iv: "af747955f5858487f59c9b710b459615"
server_key_version: "encryptionkey@0"
I20260620 04:27:31.241072 13718 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260620 04:27:31.243386 13733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:27:31.244705 13718 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:27:31.244822 13718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929496330398-392-0/minicluster-data/master-0/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929496330398-392-0/minicluster-data/master-0/wal
uuid: "31cb951a80fc476f97c0a9298b27f27e"
format_stamp: "Formatted at 2026-06-20 04:27:31 on dist-test-slave-nk6d"
server_key: "e8cf6c9e9babbc3aa373248a070fd25a"
server_key_iv: "af747955f5858487f59c9b710b459615"
server_key_version: "encryptionkey@0"
I20260620 04:27:31.244915 13718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929496330398-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929496330398-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929496330398-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:27:31.257539 13718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:27:31.260484 13718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:27:31.260686 13718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:27:31.268280 13718 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34529
I20260620 04:27:31.268338 13785 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34529 every 8 connection(s)
I20260620 04:27:31.269271 13718 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:27:31.272064 13786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:27:31.275636 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 13718
I20260620 04:27:31.275743 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929496330398-392-0/minicluster-data/master-0/wal/instance
I20260620 04:27:31.276039 392 external_mini_cluster.cc:1428] Setting key c2e546b4b181961089590ea02d25f870
I20260620 04:27:31.275813 13786 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34529 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Jun 20 04:27:31 dist-test-slave-nk6d krb5kdc[13705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929651, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:27:31.290979 13790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:31.279058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55360 (local address 127.0.98.62:34529)
0620 04:27:31.279407 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0620 04:27:31.279417 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 04:27:31.279458 (+ 41us) server_negotiation.cc:407] Connection header received
0620 04:27:31.280003 (+ 545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:27:31.280009 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:27:31.280206 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:27:31.280373 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:27:31.281188 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:31.282079 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:31.283086 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:31.283458 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:31.283605 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:27:31.285987 (+ 2382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:27:31.285993 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:27:31.286005 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:27:31.286026 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:27:31.287714 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:27:31.288196 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:27:31.288202 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:27:31.288204 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:27:31.288310 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:27:31.288588 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:27:31.288592 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:27:31.288593 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:27:31.288987 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:27:31.289117 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 04:27:31.289472 (+ 355us) server_negotiation.cc:299] Negotiation successful
0620 04:27:31.289856 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":91,"threads_started":1}
I20260620 04:27:31.290977 13789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:31.278181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34529 (local address 127.0.0.1:55360)
0620 04:27:31.278662 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:31.278714 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 04:27:31.279687 (+ 973us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:27:31.280379 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:27:31.280384 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:27:31.280812 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:27:31.281051 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:27:31.281065 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:27:31.282201 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:27:31.282204 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:27:31.282981 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:27:31.282990 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:27:31.283601 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:27:31.283605 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:27:31.283698 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:27:31.284324 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:27:31.284341 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:27:31.285843 (+ 1502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:27:31.287814 (+ 1971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:27:31.287822 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:27:31.287834 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:27:31.288102 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:27:31.288390 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:27:31.288393 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:27:31.288396 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:27:31.288502 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:27:31.289091 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:27:31.289098 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:27:31.289250 (+ 152us) client_negotiation.cc:769] Sending connection context
0620 04:27:31.289402 (+ 152us) client_negotiation.cc:240] Negotiation successful
0620 04:27:31.289927 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"mutex_wait_us":26,"thread_start_us":118,"threads_started":1}
I20260620 04:27:31.293776 13786 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37441 } has no permanent_uuid. Determining permanent_uuid...
I20260620 04:27:31.294590 13789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:31.294308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37441 (local address 127.0.0.1:57784)
0620 04:27:31.294441 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:31.294518 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 04:27:31.294836 13735 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37441: connect: Connection refused (error 111)
I20260620 04:27:31.295634 13789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:31.295434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37441 (local address 127.0.0.1:57794)
0620 04:27:31.295531 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:31.295578 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260620 04:27:31.295778 13786 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:37441: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37441: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929496330398-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929496330398-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260620 04:27:31.298928 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929496330398-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929496330398-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929496330398-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929496330398-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:37441
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929496330398-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34529,127.0.98.61:37441,127.0.98.60:37985
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 04:27:31.345054 13786 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37441 } attempt: 1
I20260620 04:27:31.345911 13789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:31.345630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37441 (local address 127.0.0.1:57796)
0620 04:27:31.345770 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:31.345836 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260620 04:27:31.346541 13789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:31.346367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37441 (local address 127.0.0.1:57802)
0620 04:27:31.346446 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:31.346489 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260620 04:27:31.346664 13786 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:37441: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37441: connect: Connection refused (error 111)
W20260620 04:27:31.409379 13793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:27:31.409679 13793 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:27:31.409722 13793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:27:31.413947 13793 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:27:31.414013 13793 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:27:31.414031 13793 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:27:31.414047 13793 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:27:31.414062 13793 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:27:31.414809 13786 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37441 } attempt: 2
I20260620 04:27:31.415619 13789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:31.415335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37441 (local address 127.0.0.1:57804)
0620 04:27:31.415480 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:31.415542 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260620 04:27:31.416353 13789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:31.416149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37441 (local address 127.0.0.1:57806)
0620 04:27:31.416243 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:31.416295 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260620 04:27:31.416463 13786 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:37441: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37441: connect: Connection refused (error 111)
I20260620 04:27:31.419849 13793 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929496330398-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929496330398-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34529,127.0.98.61:37441,127.0.98.60:37985
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929496330398-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:37441
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929496330398-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13793
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929496330398-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:27:31.420970 13793 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:27:31.422017 13793 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:27:31.430665 13801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:27:31.430665 13798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:27:31.430665 13799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:27:31.431043 13793 server_base.cc:1034] running on GCE node
I20260620 04:27:31.431638 13793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:27:31.432860 13793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:27:31.434015 13793 hybrid_clock.cc:648] HybridClock initialized: now 1781929651433967 us; error 70 us; skew 500 ppm
Jun 20 04:27:31 dist-test-slave-nk6d krb5kdc[13705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929651, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:27:31.436744 13793 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260620 04:27:31.437733 13793 webserver.cc:460] Webserver started at http://127.0.98.61:34597/ using document root <none> and password file <none>
I20260620 04:27:31.438269 13793 fs_manager.cc:359] Metadata directory not provided
I20260620 04:27:31.438344 13793 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:27:31.438534 13793 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:27:31.440701 13793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929496330398-392-0/minicluster-data/master-1/data/instance:
uuid: "bcbfd113b70c43bf986f91ee121fa456"
format_stamp: "Formatted at 2026-06-20 04:27:31 on dist-test-slave-nk6d"
server_key: "ea19080d1ae63fa5c39ca7778174a9de"
server_key_iv: "efbb9010480fb822bbbe6ca010f96abb"
server_key_version: "encryptionkey@0"
I20260620 04:27:31.441243 13793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929496330398-392-0/minicluster-data/master-1/wal/instance:
uuid: "bcbfd113b70c43bf986f91ee121fa456"
format_stamp: "Formatted at 2026-06-20 04:27:31 on dist-test-slave-nk6d"
server_key: "ea19080d1ae63fa5c39ca7778174a9de"
server_key_iv: "efbb9010480fb822bbbe6ca010f96abb"
server_key_version: "encryptionkey@0"
I20260620 04:27:31.445003 13793 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 04:27:31.447350 13808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:27:31.448588 13793 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 04:27:31.448719 13793 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929496330398-392-0/minicluster-data/master-1/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929496330398-392-0/minicluster-data/master-1/wal
uuid: "bcbfd113b70c43bf986f91ee121fa456"
format_stamp: "Formatted at 2026-06-20 04:27:31 on dist-test-slave-nk6d"
server_key: "ea19080d1ae63fa5c39ca7778174a9de"
server_key_iv: "efbb9010480fb822bbbe6ca010f96abb"
server_key_version: "encryptionkey@0"
I20260620 04:27:31.448844 13793 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929496330398-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929496330398-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929496330398-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:27:31.463321 13793 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:27:31.466446 13793 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:27:31.466624 13793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:27:31.473488 13793 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:37441
I20260620 04:27:31.473503 13860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:37441 every 8 connection(s)
I20260620 04:27:31.474450 13793 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929496330398-392-0/minicluster-data/master-1/data/info.pb
I20260620 04:27:31.474952 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 13793
I20260620 04:27:31.475073 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929496330398-392-0/minicluster-data/master-1/wal/instance
I20260620 04:27:31.475409 392 external_mini_cluster.cc:1428] Setting key c033222730cc158fe9b68d5dab5e83f4
I20260620 04:27:31.477443 13861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:27:31.481261 13861 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34529 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Jun 20 04:27:31 dist-test-slave-nk6d krb5kdc[13705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929651, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:27:31.494983 13790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:31.483135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55368 (local address 127.0.98.62:34529)
0620 04:27:31.483302 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 04:27:31.483306 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:27:31.484230 (+ 924us) server_negotiation.cc:407] Connection header received
0620 04:27:31.485391 (+ 1161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:27:31.485393 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:27:31.485446 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:27:31.485522 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:27:31.486804 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:31.487399 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:31.488581 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:31.488817 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:31.488905 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:27:31.491340 (+ 2435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:27:31.491343 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:27:31.491346 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:27:31.491371 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:27:31.492640 (+ 1269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:27:31.493259 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:27:31.493262 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:27:31.493263 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:27:31.493330 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:27:31.493805 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:27:31.493807 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:27:31.493808 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:27:31.493973 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:27:31.494040 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 04:27:31.494740 (+ 700us) server_negotiation.cc:299] Negotiation successful
0620 04:27:31.494857 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 04:27:31.495816 13864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:31.483480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34529 (local address 127.0.0.1:55368)
0620 04:27:31.484045 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:31.484107 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 04:27:31.485182 (+ 1075us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:27:31.485754 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:27:31.485764 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:27:31.486099 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:27:31.486658 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:27:31.486671 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:27:31.487528 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:27:31.487531 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:27:31.488465 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:27:31.488475 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:27:31.488957 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:27:31.488961 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:27:31.489118 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:27:31.489660 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:27:31.489676 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:27:31.491156 (+ 1480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:27:31.492752 (+ 1596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:27:31.492758 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:27:31.492776 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:27:31.493114 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:27:31.493445 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:27:31.493449 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:27:31.493451 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:27:31.493686 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:27:31.494070 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:27:31.494077 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:27:31.494459 (+ 382us) client_negotiation.cc:769] Sending connection context
0620 04:27:31.494696 (+ 237us) client_negotiation.cc:240] Negotiation successful
0620 04:27:31.494987 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":129,"threads_started":1}
I20260620 04:27:31.497257 13861 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37441 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 04:27:31 dist-test-slave-nk6d krb5kdc[13705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929651, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929496330398-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929496330398-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260620 04:27:31.504326 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929496330398-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929496330398-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929496330398-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929496330398-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:37985
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929496330398-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34529,127.0.98.61:37441,127.0.98.60:37985
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 04:27:31.506085 13864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:31.497781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37441 (local address 127.0.0.1:57816)
0620 04:27:31.497903 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:31.497915 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 04:27:31.497979 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:27:31.498469 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:27:31.498471 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:27:31.498641 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:27:31.498716 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:27:31.498721 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:27:31.499710 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:27:31.499714 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:27:31.500352 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:27:31.500361 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:27:31.500970 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:27:31.500973 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:27:31.501058 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:27:31.501540 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:27:31.501553 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:27:31.502747 (+ 1194us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:27:31.504609 (+ 1862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:27:31.504612 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:27:31.504614 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:27:31.504791 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:27:31.505039 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:27:31.505042 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:27:31.505044 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:27:31.505110 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:27:31.505654 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:27:31.505657 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:27:31.505744 (+ 87us) client_negotiation.cc:769] Sending connection context
0620 04:27:31.505837 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 04:27:31.505949 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 04:27:31.506284 13866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:31.497873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57816 (local address 127.0.98.61:37441)
0620 04:27:31.498138 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0620 04:27:31.498147 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 04:27:31.498164 (+ 17us) server_negotiation.cc:407] Connection header received
0620 04:27:31.498193 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:27:31.498197 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:27:31.498362 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:27:31.498476 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:27:31.498808 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:31.499619 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:31.500469 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:31.500864 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:31.500974 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:27:31.502895 (+ 1921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:27:31.502904 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:27:31.502918 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:27:31.502939 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:27:31.504510 (+ 1571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:27:31.504886 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:27:31.504892 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:27:31.504893 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:27:31.504963 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:27:31.505182 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:27:31.505185 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:27:31.505200 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:27:31.505541 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:27:31.505675 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 04:27:31.505954 (+ 279us) server_negotiation.cc:299] Negotiation successful
0620 04:27:31.506108 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":89,"threads_started":1}
I20260620 04:27:31.507618 13786 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37441 } attempt: 3
I20260620 04:27:31.508091 13861 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37985 } has no permanent_uuid. Determining permanent_uuid...
I20260620 04:27:31.508916 13864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:31.508655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37985 (local address 127.0.0.1:44966)
0620 04:27:31.508753 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:31.508844 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260620 04:27:31.509683 13810 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37985: connect: Connection refused (error 111)
I20260620 04:27:31.510483 13864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:31.510270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37985 (local address 127.0.0.1:44970)
0620 04:27:31.510364 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:31.510412 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260620 04:27:31.511515 13861 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37985: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37985: connect: Connection refused (error 111)
Jun 20 04:27:31 dist-test-slave-nk6d krb5kdc[13705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929651, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 04:27:31.516889 13789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:31.508123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37441 (local address 127.0.0.1:57830)
0620 04:27:31.508238 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:31.508252 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:27:31.508330 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:27:31.508856 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:27:31.508859 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:27:31.509046 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:27:31.509495 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:27:31.509502 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:27:31.510551 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:27:31.510555 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:27:31.511398 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:27:31.511410 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:27:31.512240 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:27:31.512243 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:27:31.512338 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:27:31.512767 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:27:31.512778 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:27:31.513885 (+ 1107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:27:31.515578 (+ 1693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:27:31.515581 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:27:31.515582 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:27:31.515763 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:27:31.516051 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:27:31.516053 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:27:31.516055 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:27:31.516125 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:27:31.516506 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:27:31.516508 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:27:31.516591 (+ 83us) client_negotiation.cc:769] Sending connection context
0620 04:27:31.516664 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 04:27:31.516768 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 04:27:31.516994 13866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:31.508187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57830 (local address 127.0.98.61:37441)
0620 04:27:31.508605 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0620 04:27:31.508609 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:27:31.508623 (+ 14us) server_negotiation.cc:407] Connection header received
0620 04:27:31.508655 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:27:31.508658 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:27:31.508702 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:27:31.508830 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:27:31.509612 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:31.510410 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:31.511746 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:31.512129 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:31.512244 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:27:31.514014 (+ 1770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:27:31.514018 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:27:31.514022 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:27:31.514049 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:27:31.515474 (+ 1425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:27:31.515847 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:27:31.515851 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:27:31.515853 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:27:31.515944 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:27:31.516194 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:27:31.516197 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:27:31.516210 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:27:31.516432 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:27:31.516523 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 04:27:31.516731 (+ 208us) server_negotiation.cc:299] Negotiation successful
0620 04:27:31.516847 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345}
I20260620 04:27:31.517644 13786 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37985 } has no permanent_uuid. Determining permanent_uuid...
I20260620 04:27:31.518285 13789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:31.518061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37985 (local address 127.0.0.1:44974)
0620 04:27:31.518175 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:31.518225 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260620 04:27:31.518854 13789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:31.518630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37985 (local address 127.0.0.1:44976)
0620 04:27:31.518758 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:31.518808 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260620 04:27:31.518965 13786 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37985: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37985: connect: Connection refused (error 111)
I20260620 04:27:31.550104 13786 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37985 } attempt: 1
I20260620 04:27:31.550976 13789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:31.550664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37985 (local address 127.0.0.1:44978)
0620 04:27:31.550812 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:31.550879 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260620 04:27:31.551693 13789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:31.551461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37985 (local address 127.0.0.1:44980)
0620 04:27:31.551581 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:31.551630 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260620 04:27:31.551923 13786 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37985: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37985: connect: Connection refused (error 111)
I20260620 04:27:31.560794 13861 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37985 } attempt: 1
I20260620 04:27:31.561676 13864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:31.561321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37985 (local address 127.0.0.1:44994)
0620 04:27:31.561482 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:31.561581 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260620 04:27:31.562366 13864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:31.562142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37985 (local address 127.0.0.1:44996)
0620 04:27:31.562250 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:31.562303 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260620 04:27:31.562505 13861 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37985: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37985: connect: Connection refused (error 111)
W20260620 04:27:31.621050 13867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:27:31.621344 13867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:27:31.621398 13867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:27:31.625892 13867 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:27:31.625967 13867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:27:31.625986 13867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:27:31.626003 13867 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:27:31.626017 13867 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:27:31.627116 13786 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37985 } attempt: 2
I20260620 04:27:31.627899 13789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:31.627616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37985 (local address 127.0.0.1:45006)
0620 04:27:31.627760 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:31.627821 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260620 04:27:31.628604 13789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:31.628417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37985 (local address 127.0.0.1:45020)
0620 04:27:31.628503 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:31.628549 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260620 04:27:31.628727 13786 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37985: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37985: connect: Connection refused (error 111)
I20260620 04:27:31.630661 13861 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37985 } attempt: 2
I20260620 04:27:31.631350 13864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:31.631060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37985 (local address 127.0.0.1:45024)
0620 04:27:31.631204 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:31.631266 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260620 04:27:31.631983 13864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:31.631765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37985 (local address 127.0.0.1:45028)
0620 04:27:31.631867 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:31.631907 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260620 04:27:31.632097 13861 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37985: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37985: connect: Connection refused (error 111)
I20260620 04:27:31.632302 13867 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929496330398-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929496330398-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34529,127.0.98.61:37441,127.0.98.60:37985
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929496330398-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:37985
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929496330398-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13867
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929496330398-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:27:31.633456 13867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:27:31.634502 13867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:27:31.642724 13876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:27:31.642724 13874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:27:31.642745 13873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:27:31.643115 13867 server_base.cc:1034] running on GCE node
I20260620 04:27:31.643778 13867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:27:31.644965 13867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:27:31.646129 13867 hybrid_clock.cc:648] HybridClock initialized: now 1781929651646111 us; error 37 us; skew 500 ppm
Jun 20 04:27:31 dist-test-slave-nk6d krb5kdc[13705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929651, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:27:31.648888 13867 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260620 04:27:31.649770 13867 webserver.cc:460] Webserver started at http://127.0.98.60:32939/ using document root <none> and password file <none>
I20260620 04:27:31.650293 13867 fs_manager.cc:359] Metadata directory not provided
I20260620 04:27:31.650364 13867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:27:31.650552 13867 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:27:31.652650 13867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929496330398-392-0/minicluster-data/master-2/data/instance:
uuid: "62546b3a10ed4f62818cf3016b3a0d9c"
format_stamp: "Formatted at 2026-06-20 04:27:31 on dist-test-slave-nk6d"
server_key: "ca1519a965d3a15c7ce35bb06c30be80"
server_key_iv: "371c5e3fce02ee3bda27862376be4799"
server_key_version: "encryptionkey@0"
I20260620 04:27:31.653187 13867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929496330398-392-0/minicluster-data/master-2/wal/instance:
uuid: "62546b3a10ed4f62818cf3016b3a0d9c"
format_stamp: "Formatted at 2026-06-20 04:27:31 on dist-test-slave-nk6d"
server_key: "ca1519a965d3a15c7ce35bb06c30be80"
server_key_iv: "371c5e3fce02ee3bda27862376be4799"
server_key_version: "encryptionkey@0"
I20260620 04:27:31.656762 13867 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 04:27:31.659076 13883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:27:31.660337 13867 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:27:31.660446 13867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929496330398-392-0/minicluster-data/master-2/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929496330398-392-0/minicluster-data/master-2/wal
uuid: "62546b3a10ed4f62818cf3016b3a0d9c"
format_stamp: "Formatted at 2026-06-20 04:27:31 on dist-test-slave-nk6d"
server_key: "ca1519a965d3a15c7ce35bb06c30be80"
server_key_iv: "371c5e3fce02ee3bda27862376be4799"
server_key_version: "encryptionkey@0"
I20260620 04:27:31.660533 13867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929496330398-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929496330398-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929496330398-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:27:31.682793 13867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:27:31.685932 13867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:27:31.686117 13867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:27:31.692791 13867 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:37985
I20260620 04:27:31.692822 13935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:37985 every 8 connection(s)
I20260620 04:27:31.693722 13867 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929496330398-392-0/minicluster-data/master-2/data/info.pb
I20260620 04:27:31.696507 13936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:27:31.700501 13936 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34529 } has no permanent_uuid. Determining permanent_uuid...
I20260620 04:27:31.701705 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 13867
I20260620 04:27:31.701843 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929496330398-392-0/minicluster-data/master-2/wal/instance
I20260620 04:27:31.702147 392 external_mini_cluster.cc:1428] Setting key e03f33834ff98b7656c9719a461a94aa
Jun 20 04:27:31 dist-test-slave-nk6d krb5kdc[13705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929651, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Jun 20 04:27:31 dist-test-slave-nk6d krb5kdc[13705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929651, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 04:27:31.713703 13790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:31.702567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55380 (local address 127.0.98.62:34529)
0620 04:27:31.702729 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 04:27:31.702734 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:27:31.703543 (+ 809us) server_negotiation.cc:407] Connection header received
0620 04:27:31.704635 (+ 1092us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:27:31.704638 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:27:31.704689 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:27:31.704762 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:27:31.705664 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:31.706160 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:31.707237 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:31.707531 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:31.707608 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:27:31.710151 (+ 2543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:27:31.710156 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:27:31.710159 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:27:31.710190 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:27:31.711641 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:27:31.712157 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:27:31.712161 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:27:31.712162 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:27:31.712247 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:27:31.712562 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:27:31.712565 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:27:31.712567 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:27:31.712756 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:27:31.712839 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 04:27:31.713273 (+ 434us) server_negotiation.cc:299] Negotiation successful
0620 04:27:31.713391 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 04:27:31.714393 13939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:31.702955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34529 (local address 127.0.0.1:55380)
0620 04:27:31.703412 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:31.703453 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 04:27:31.704406 (+ 953us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:27:31.704828 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:27:31.704833 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:27:31.705266 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:27:31.705516 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:27:31.705535 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:27:31.706258 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:27:31.706262 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:27:31.707118 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:27:31.707127 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:27:31.707669 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:27:31.707673 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:27:31.707833 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:27:31.708498 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:27:31.708514 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:27:31.709962 (+ 1448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:27:31.711744 (+ 1782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:27:31.711753 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:27:31.711763 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:27:31.712071 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:27:31.712328 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:27:31.712331 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:27:31.712333 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:27:31.712468 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:27:31.712852 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:27:31.712859 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:27:31.713032 (+ 173us) client_negotiation.cc:769] Sending connection context
0620 04:27:31.713171 (+ 139us) client_negotiation.cc:240] Negotiation successful
0620 04:27:31.713490 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":122,"threads_started":1}
I20260620 04:27:31.714406 13940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:31.703714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45030 (local address 127.0.98.60:37985)
0620 04:27:31.703984 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0620 04:27:31.703995 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 04:27:31.704038 (+ 43us) server_negotiation.cc:407] Connection header received
0620 04:27:31.704412 (+ 374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:27:31.704435 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:27:31.704674 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:27:31.704849 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:27:31.705545 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:31.706547 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:31.707449 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:31.707921 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:31.708039 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:27:31.710528 (+ 2489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:27:31.710534 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:27:31.710545 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:27:31.710564 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:27:31.711920 (+ 1356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:27:31.712309 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:27:31.712315 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:27:31.712317 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:27:31.712433 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:27:31.712677 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:27:31.712680 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:27:31.712682 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:27:31.713024 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:27:31.713199 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0620 04:27:31.713418 (+ 219us) server_negotiation.cc:299] Negotiation successful
0620 04:27:31.713549 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":69,"threads_started":1}
I20260620 04:27:31.715809 13936 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37441 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 04:27:31 dist-test-slave-nk6d krb5kdc[13705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929651, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Jun 20 04:27:31 dist-test-slave-nk6d krb5kdc[13705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929651, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:27:31.723263 13861 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37985 } attempt: 3
I20260620 04:27:31.723752 13939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:31.716325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37441 (local address 127.0.0.1:57834)
0620 04:27:31.716437 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:31.716451 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:27:31.716646 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:27:31.716901 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:27:31.716903 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:27:31.717034 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:27:31.717127 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:27:31.717132 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:27:31.717839 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:27:31.717842 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:27:31.718346 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:27:31.718353 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:27:31.718739 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:27:31.718743 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:27:31.718848 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:27:31.719296 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:27:31.719310 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:27:31.720666 (+ 1356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:27:31.722313 (+ 1647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:27:31.722317 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:27:31.722319 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:27:31.722536 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:27:31.722810 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:27:31.722813 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:27:31.722816 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:27:31.722907 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:27:31.723255 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:27:31.723258 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:27:31.723358 (+ 100us) client_negotiation.cc:769] Sending connection context
0620 04:27:31.723462 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 04:27:31.723586 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260620 04:27:31.723764 13866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:31.716453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57834 (local address 127.0.98.61:37441)
0620 04:27:31.716568 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 04:27:31.716572 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:27:31.716583 (+ 11us) server_negotiation.cc:407] Connection header received
0620 04:27:31.716747 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:27:31.716750 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:27:31.716798 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:27:31.716874 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:27:31.717238 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:31.717736 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:31.718444 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:31.718651 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:31.718734 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:27:31.720858 (+ 2124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:27:31.720860 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:27:31.720863 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:27:31.720886 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:27:31.722212 (+ 1326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:27:31.722645 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:27:31.722648 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:27:31.722649 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:27:31.722728 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:27:31.722984 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:27:31.722986 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:27:31.722988 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:27:31.723146 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:27:31.723256 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 04:27:31.723517 (+ 261us) server_negotiation.cc:299] Negotiation successful
0620 04:27:31.723627 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 04:27:31.724789 13936 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37985 } has no permanent_uuid. Determining permanent_uuid...
I20260620 04:27:31.724994 13790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:31.716746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55394 (local address 127.0.98.62:34529)
0620 04:27:31.716865 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 04:27:31.716869 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:27:31.717089 (+ 220us) server_negotiation.cc:407] Connection header received
0620 04:27:31.717177 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:27:31.717180 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:27:31.717221 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:27:31.717284 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:27:31.717845 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:31.718389 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:31.719137 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:31.719434 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:31.719532 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:27:31.721779 (+ 2247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:27:31.721783 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:27:31.721786 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:27:31.721810 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:27:31.723188 (+ 1378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:27:31.723627 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:27:31.723630 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:27:31.723632 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:27:31.723710 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:27:31.724013 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:27:31.724016 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:27:31.724017 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:27:31.724211 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:27:31.724400 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0620 04:27:31.724754 (+ 354us) server_negotiation.cc:299] Negotiation successful
0620 04:27:31.724861 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 04:27:31.727870 13786 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37985 } attempt: 3
Jun 20 04:27:31 dist-test-slave-nk6d krb5kdc[13705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929651, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 20 04:27:31 dist-test-slave-nk6d krb5kdc[13705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929651, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Jun 20 04:27:31 dist-test-slave-nk6d krb5kdc[13705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929651, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 04:27:31.733826 13864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:31.723633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37985 (local address 127.0.0.1:45040)
0620 04:27:31.723907 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:31.723920 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:27:31.724029 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:27:31.725172 (+ 1143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:27:31.725174 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:27:31.725334 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:27:31.725422 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:27:31.725429 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:27:31.726525 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:27:31.726528 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:27:31.727430 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:27:31.727441 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:27:31.728044 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:27:31.728048 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:27:31.728134 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:27:31.728835 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:27:31.728848 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:27:31.730222 (+ 1374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:27:31.732251 (+ 2029us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:27:31.732254 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:27:31.732257 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:27:31.732491 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:27:31.732762 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:27:31.732764 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:27:31.732766 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:27:31.732856 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:27:31.733289 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:27:31.733292 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:27:31.733395 (+ 103us) client_negotiation.cc:769] Sending connection context
0620 04:27:31.733508 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 04:27:31.733642 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"mutex_wait_us":26}
I20260620 04:27:31.733861 13940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:31.724164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45040 (local address 127.0.98.60:37985)
0620 04:27:31.724989 (+ 825us) server_negotiation.cc:206] Beginning negotiation
0620 04:27:31.724993 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:27:31.725004 (+ 11us) server_negotiation.cc:407] Connection header received
0620 04:27:31.725036 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:27:31.725038 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:27:31.725075 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:27:31.725145 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:27:31.725543 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:31.726441 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:31.727542 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:31.727893 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:31.728745 (+ 852us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:27:31.730339 (+ 1594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:27:31.730342 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:27:31.730344 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:27:31.730364 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:27:31.732128 (+ 1764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:27:31.732592 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:27:31.732595 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:27:31.732597 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:27:31.732673 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:27:31.732947 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:27:31.732950 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:27:31.732952 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:27:31.733163 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:27:31.733296 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 04:27:31.733572 (+ 276us) server_negotiation.cc:299] Negotiation successful
0620 04:27:31.733700 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":752}
Jun 20 04:27:31 dist-test-slave-nk6d krb5kdc[13705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929651, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 04:27:31.735560 13939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:31.725438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37985 (local address 127.0.0.1:45042)
0620 04:27:31.725525 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:31.725537 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 04:27:31.725595 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:27:31.726101 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:27:31.726103 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:27:31.726241 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:27:31.726302 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:27:31.726307 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:27:31.727768 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:27:31.727771 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:27:31.728561 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:27:31.728572 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:27:31.729099 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:27:31.729103 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:27:31.729194 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:27:31.730355 (+ 1161us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:27:31.730380 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:27:31.732274 (+ 1894us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:27:31.734100 (+ 1826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:27:31.734103 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:27:31.734105 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:27:31.734315 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:27:31.734549 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:27:31.734552 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:27:31.734554 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:27:31.734645 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:27:31.735097 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:27:31.735100 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:27:31.735182 (+ 82us) client_negotiation.cc:769] Sending connection context
0620 04:27:31.735269 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 04:27:31.735389 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26,"spinlock_wait_cycles":1536}
I20260620 04:27:31.736055 13941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:31.725581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45042 (local address 127.0.98.60:37985)
0620 04:27:31.725851 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0620 04:27:31.725859 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 04:27:31.725872 (+ 13us) server_negotiation.cc:407] Connection header received
0620 04:27:31.725902 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:27:31.725904 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:27:31.725941 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:27:31.726011 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:27:31.726456 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:31.727662 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:31.728657 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:31.728991 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:31.729077 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:27:31.732405 (+ 3328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:27:31.732408 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:27:31.732410 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:27:31.732431 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:27:31.733990 (+ 1559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:27:31.734396 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:27:31.734398 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:27:31.734400 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:27:31.734467 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:27:31.734744 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:27:31.734746 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:27:31.734748 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:27:31.734981 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:27:31.735065 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 04:27:31.735757 (+ 692us) server_negotiation.cc:299] Negotiation successful
0620 04:27:31.735863 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":99,"threads_started":1}
I20260620 04:27:31.737655 13866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:31.726103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57844 (local address 127.0.98.61:37441)
0620 04:27:31.726196 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0620 04:27:31.726200 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:27:31.726224 (+ 24us) server_negotiation.cc:407] Connection header received
0620 04:27:31.726343 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:27:31.726346 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:27:31.726383 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:27:31.726981 (+ 598us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:27:31.727056 (+ 75us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:31.727567 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:31.728556 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:31.728873 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:31.728956 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:27:31.732414 (+ 3458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:27:31.732417 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:27:31.732420 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:27:31.732443 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:27:31.735054 (+ 2611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:27:31.736063 (+ 1009us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:27:31.736066 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:27:31.736068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:27:31.736474 (+ 406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:27:31.736787 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:27:31.736789 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:27:31.736790 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:27:31.736951 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:27:31.737160 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0620 04:27:31.737313 (+ 153us) server_negotiation.cc:299] Negotiation successful
0620 04:27:31.737417 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":40}
I20260620 04:27:31.737876 13861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bcbfd113b70c43bf986f91ee121fa456: Bootstrap starting.
I20260620 04:27:31.738955 13789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:31.728256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37985 (local address 127.0.0.1:45050)
0620 04:27:31.728467 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:31.728480 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:27:31.728543 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:27:31.729820 (+ 1277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:27:31.729824 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:27:31.730072 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:27:31.730244 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:27:31.730252 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:27:31.731564 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:27:31.731568 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:27:31.732390 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:27:31.732401 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:27:31.733601 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:27:31.733605 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:27:31.733697 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:27:31.734088 (+ 391us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:27:31.734102 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:27:31.735408 (+ 1306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:27:31.737527 (+ 2119us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:27:31.737530 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:27:31.737532 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:27:31.737768 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:27:31.738056 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:27:31.738059 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:27:31.738061 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:27:31.738140 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:27:31.738541 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:27:31.738543 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:27:31.738629 (+ 86us) client_negotiation.cc:769] Sending connection context
0620 04:27:31.738732 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 04:27:31.738842 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145}
I20260620 04:27:31.739076 13942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:31.728508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45050 (local address 127.0.98.60:37985)
0620 04:27:31.729537 (+ 1029us) server_negotiation.cc:206] Beginning negotiation
0620 04:27:31.729545 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 04:27:31.729556 (+ 11us) server_negotiation.cc:407] Connection header received
0620 04:27:31.729586 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:27:31.729589 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:27:31.729625 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:27:31.729770 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:27:31.730431 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:31.731464 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:31.732517 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:31.733175 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:31.735538 (+ 2363us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:27:31.735638 (+ 100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:27:31.735641 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:27:31.735643 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:27:31.735658 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:27:31.737403 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:27:31.737889 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:27:31.737892 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:27:31.737894 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:27:31.737972 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:27:31.738234 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:27:31.738237 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:27:31.738238 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:27:31.738440 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:27:31.738535 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 04:27:31.738798 (+ 263us) server_negotiation.cc:299] Negotiation successful
0620 04:27:31.738906 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":967,"thread_start_us":96,"threads_started":1}
I20260620 04:27:31.742324 13786 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31cb951a80fc476f97c0a9298b27f27e: Bootstrap starting.
I20260620 04:27:31.742391 13861 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bcbfd113b70c43bf986f91ee121fa456: Neither blocks nor log segments found. Creating new log.
I20260620 04:27:31.742588 13936 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62546b3a10ed4f62818cf3016b3a0d9c: Bootstrap starting.
I20260620 04:27:31.743422 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 04:27:31.744911 13936 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 62546b3a10ed4f62818cf3016b3a0d9c: Neither blocks nor log segments found. Creating new log.
I20260620 04:27:31.745105 13786 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 31cb951a80fc476f97c0a9298b27f27e: Neither blocks nor log segments found. Creating new log.
I20260620 04:27:31.745620 13936 log.cc:824] T 00000000000000000000000000000000 P 62546b3a10ed4f62818cf3016b3a0d9c: Log is configured to *not* fsync() on all Append() calls
I20260620 04:27:31.745858 13786 log.cc:824] T 00000000000000000000000000000000 P 31cb951a80fc476f97c0a9298b27f27e: Log is configured to *not* fsync() on all Append() calls
I20260620 04:27:31.745889 13861 log.cc:824] T 00000000000000000000000000000000 P bcbfd113b70c43bf986f91ee121fa456: Log is configured to *not* fsync() on all Append() calls
I20260620 04:27:31.747952 13861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bcbfd113b70c43bf986f91ee121fa456: No bootstrap required, opened a new log
I20260620 04:27:31.747951 13786 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31cb951a80fc476f97c0a9298b27f27e: No bootstrap required, opened a new log
I20260620 04:27:31.748401 13936 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62546b3a10ed4f62818cf3016b3a0d9c: No bootstrap required, opened a new log
I20260620 04:27:31.752221 13861 raft_consensus.cc:348] T 00000000000000000000000000000000 P bcbfd113b70c43bf986f91ee121fa456 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31cb951a80fc476f97c0a9298b27f27e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34529 } } peers { permanent_uuid: "bcbfd113b70c43bf986f91ee121fa456" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37441 } } peers { permanent_uuid: "62546b3a10ed4f62818cf3016b3a0d9c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37985 } }
I20260620 04:27:31.752311 13786 raft_consensus.cc:348] T 00000000000000000000000000000000 P 31cb951a80fc476f97c0a9298b27f27e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31cb951a80fc476f97c0a9298b27f27e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34529 } } peers { permanent_uuid: "bcbfd113b70c43bf986f91ee121fa456" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37441 } } peers { permanent_uuid: "62546b3a10ed4f62818cf3016b3a0d9c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37985 } }
I20260620 04:27:31.752534 13786 raft_consensus.cc:374] T 00000000000000000000000000000000 P 31cb951a80fc476f97c0a9298b27f27e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:27:31.752583 13786 raft_consensus.cc:729] T 00000000000000000000000000000000 P 31cb951a80fc476f97c0a9298b27f27e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31cb951a80fc476f97c0a9298b27f27e, State: Initialized, Role: FOLLOWER
I20260620 04:27:31.752687 13861 raft_consensus.cc:374] T 00000000000000000000000000000000 P bcbfd113b70c43bf986f91ee121fa456 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:27:31.752795 13861 raft_consensus.cc:729] T 00000000000000000000000000000000 P bcbfd113b70c43bf986f91ee121fa456 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcbfd113b70c43bf986f91ee121fa456, State: Initialized, Role: FOLLOWER
I20260620 04:27:31.752892 13936 raft_consensus.cc:348] T 00000000000000000000000000000000 P 62546b3a10ed4f62818cf3016b3a0d9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31cb951a80fc476f97c0a9298b27f27e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34529 } } peers { permanent_uuid: "bcbfd113b70c43bf986f91ee121fa456" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37441 } } peers { permanent_uuid: "62546b3a10ed4f62818cf3016b3a0d9c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37985 } }
I20260620 04:27:31.753113 13936 raft_consensus.cc:374] T 00000000000000000000000000000000 P 62546b3a10ed4f62818cf3016b3a0d9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:27:31.753165 13936 raft_consensus.cc:729] T 00000000000000000000000000000000 P 62546b3a10ed4f62818cf3016b3a0d9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62546b3a10ed4f62818cf3016b3a0d9c, State: Initialized, Role: FOLLOWER
I20260620 04:27:31.753167 13786 consensus_queue.cc:260] T 00000000000000000000000000000000 P 31cb951a80fc476f97c0a9298b27f27e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31cb951a80fc476f97c0a9298b27f27e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34529 } } peers { permanent_uuid: "bcbfd113b70c43bf986f91ee121fa456" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37441 } } peers { permanent_uuid: "62546b3a10ed4f62818cf3016b3a0d9c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37985 } }
I20260620 04:27:31.753567 13861 consensus_queue.cc:260] T 00000000000000000000000000000000 P bcbfd113b70c43bf986f91ee121fa456 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31cb951a80fc476f97c0a9298b27f27e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34529 } } peers { permanent_uuid: "bcbfd113b70c43bf986f91ee121fa456" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37441 } } peers { permanent_uuid: "62546b3a10ed4f62818cf3016b3a0d9c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37985 } }
I20260620 04:27:31.753608 13936 consensus_queue.cc:260] T 00000000000000000000000000000000 P 62546b3a10ed4f62818cf3016b3a0d9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31cb951a80fc476f97c0a9298b27f27e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34529 } } peers { permanent_uuid: "bcbfd113b70c43bf986f91ee121fa456" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37441 } } peers { permanent_uuid: "62546b3a10ed4f62818cf3016b3a0d9c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37985 } }
I20260620 04:27:31.754591 13786 sys_catalog.cc:564] T 00000000000000000000000000000000 P 31cb951a80fc476f97c0a9298b27f27e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:27:31.754992 13861 sys_catalog.cc:564] T 00000000000000000000000000000000 P bcbfd113b70c43bf986f91ee121fa456 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:27:31.755180 13936 sys_catalog.cc:564] T 00000000000000000000000000000000 P 62546b3a10ed4f62818cf3016b3a0d9c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:27:31.755892 13790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:31.745701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55406 (local address 127.0.98.62:34529)
0620 04:27:31.745827 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 04:27:31.745832 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:27:31.746841 (+ 1009us) server_negotiation.cc:407] Connection header received
0620 04:27:31.747017 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:27:31.747020 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:27:31.747063 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:27:31.747111 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:27:31.748149 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:31.748909 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:31.750078 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:31.750401 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:31.750476 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:27:31.752357 (+ 1881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:27:31.752361 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:27:31.752380 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:27:31.752408 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:27:31.754039 (+ 1631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:27:31.754517 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:27:31.754521 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:27:31.754523 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:27:31.754606 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:27:31.754918 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:27:31.754921 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:27:31.754923 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:27:31.755145 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:27:31.755245 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 04:27:31.755602 (+ 357us) server_negotiation.cc:299] Negotiation successful
0620 04:27:31.755723 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 04:27:31.756819 13953 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62546b3a10ed4f62818cf3016b3a0d9c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31cb951a80fc476f97c0a9298b27f27e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34529 } } peers { permanent_uuid: "bcbfd113b70c43bf986f91ee121fa456" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37441 } } peers { permanent_uuid: "62546b3a10ed4f62818cf3016b3a0d9c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37985 } } }
I20260620 04:27:31.756947 13953 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62546b3a10ed4f62818cf3016b3a0d9c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 04:27:31.757056 13953 raft_consensus.cc:482] T 00000000000000000000000000000000 P 62546b3a10ed4f62818cf3016b3a0d9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:27:31.757172 13953 raft_consensus.cc:504] T 00000000000000000000000000000000 P 62546b3a10ed4f62818cf3016b3a0d9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31cb951a80fc476f97c0a9298b27f27e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34529 } } peers { permanent_uuid: "bcbfd113b70c43bf986f91ee121fa456" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37441 } } peers { permanent_uuid: "62546b3a10ed4f62818cf3016b3a0d9c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37985 } }
I20260620 04:27:31.762156 13759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "62546b3a10ed4f62818cf3016b3a0d9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31cb951a80fc476f97c0a9298b27f27e" is_pre_election: true
I20260620 04:27:31.762494 13759 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 31cb951a80fc476f97c0a9298b27f27e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62546b3a10ed4f62818cf3016b3a0d9c in term 0.
I20260620 04:27:31.762905 13884 leader_election.cc:302] T 00000000000000000000000000000000 P 62546b3a10ed4f62818cf3016b3a0d9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31cb951a80fc476f97c0a9298b27f27e, 62546b3a10ed4f62818cf3016b3a0d9c; no voters:
I20260620 04:27:31.763252 13961 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 62546b3a10ed4f62818cf3016b3a0d9c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 04:27:31.763331 13961 raft_consensus.cc:482] T 00000000000000000000000000000000 P 62546b3a10ed4f62818cf3016b3a0d9c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 04:27:31.763371 13961 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 62546b3a10ed4f62818cf3016b3a0d9c [term 0 FOLLOWER]: Advancing to term 1
W20260620 04:27:31.764703 13739 tablet.cc:2369] T 00000000000000000000000000000000 P 31cb951a80fc476f97c0a9298b27f27e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 04:27:31.764742 13961 raft_consensus.cc:504] T 00000000000000000000000000000000 P 62546b3a10ed4f62818cf3016b3a0d9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31cb951a80fc476f97c0a9298b27f27e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34529 } } peers { permanent_uuid: "bcbfd113b70c43bf986f91ee121fa456" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37441 } } peers { permanent_uuid: "62546b3a10ed4f62818cf3016b3a0d9c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37985 } }
I20260620 04:27:31.765105 13961 leader_election.cc:288] T 00000000000000000000000000000000 P 62546b3a10ed4f62818cf3016b3a0d9c [CANDIDATE]: Term 1 election: Requested vote from peers 31cb951a80fc476f97c0a9298b27f27e (127.0.98.62:34529), bcbfd113b70c43bf986f91ee121fa456 (127.0.98.61:37441)
I20260620 04:27:31.765923 13834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "62546b3a10ed4f62818cf3016b3a0d9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcbfd113b70c43bf986f91ee121fa456" is_pre_election: true
I20260620 04:27:31.766199 13834 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bcbfd113b70c43bf986f91ee121fa456 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62546b3a10ed4f62818cf3016b3a0d9c in term 0.
I20260620 04:27:31.766360 13942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:31.746878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45052 (local address 127.0.98.60:37985)
0620 04:27:31.751778 (+ 4900us) server_negotiation.cc:206] Beginning negotiation
0620 04:27:31.751783 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:27:31.757573 (+ 5790us) server_negotiation.cc:407] Connection header received
0620 04:27:31.757737 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:27:31.757740 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:27:31.757787 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:27:31.757845 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:27:31.758673 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:31.759434 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:31.760524 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:31.760898 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:31.760970 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:27:31.762740 (+ 1770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:27:31.762744 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:27:31.762746 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:27:31.762774 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:27:31.764349 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:27:31.764973 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:27:31.764977 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:27:31.764979 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:27:31.765061 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:27:31.765355 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:27:31.765358 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:27:31.765360 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:27:31.765580 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:27:31.765715 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 04:27:31.766086 (+ 371us) server_negotiation.cc:299] Negotiation successful
0620 04:27:31.766214 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4802}
I20260620 04:27:31.766557 13953 leader_election.cc:288] T 00000000000000000000000000000000 P 62546b3a10ed4f62818cf3016b3a0d9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31cb951a80fc476f97c0a9298b27f27e (127.0.98.62:34529), bcbfd113b70c43bf986f91ee121fa456 (127.0.98.61:37441)
I20260620 04:27:31.766880 13866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:31.746537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57850 (local address 127.0.98.61:37441)
0620 04:27:31.746645 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0620 04:27:31.746649 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:27:31.757497 (+ 10848us) server_negotiation.cc:407] Connection header received
0620 04:27:31.757617 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:27:31.757621 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:27:31.757671 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:27:31.757746 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:27:31.759218 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:31.759886 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:31.761101 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:31.761430 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:31.761523 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:27:31.763349 (+ 1826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:27:31.763353 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:27:31.763356 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:27:31.763380 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:27:31.765105 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:27:31.765563 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:27:31.765566 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:27:31.765568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:27:31.765648 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:27:31.765993 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:27:31.765996 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:27:31.765998 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:27:31.766199 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:27:31.766291 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 04:27:31.766621 (+ 330us) server_negotiation.cc:299] Negotiation successful
0620 04:27:31.766732 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 04:27:31.765715 13835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "62546b3a10ed4f62818cf3016b3a0d9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcbfd113b70c43bf986f91ee121fa456"
I20260620 04:27:31.767993 13835 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bcbfd113b70c43bf986f91ee121fa456 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:27:31.769380 13835 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bcbfd113b70c43bf986f91ee121fa456 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62546b3a10ed4f62818cf3016b3a0d9c in term 1.
I20260620 04:27:31.769672 13885 leader_election.cc:302] T 00000000000000000000000000000000 P 62546b3a10ed4f62818cf3016b3a0d9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62546b3a10ed4f62818cf3016b3a0d9c, bcbfd113b70c43bf986f91ee121fa456; no voters:
W20260620 04:27:31.769917 13977 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 31cb951a80fc476f97c0a9298b27f27e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 04:27:31.770002 13977 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 04:27:31.770079 13760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "62546b3a10ed4f62818cf3016b3a0d9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31cb951a80fc476f97c0a9298b27f27e"
I20260620 04:27:31.770161 13953 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 62546b3a10ed4f62818cf3016b3a0d9c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:27:31.770198 13760 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 31cb951a80fc476f97c0a9298b27f27e [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:27:31.770357 13953 raft_consensus.cc:686] T 00000000000000000000000000000000 P 62546b3a10ed4f62818cf3016b3a0d9c [term 1 LEADER]: Becoming Leader. State: Replica: 62546b3a10ed4f62818cf3016b3a0d9c, State: Running, Role: LEADER
I20260620 04:27:31.770685 13953 consensus_queue.cc:237] T 00000000000000000000000000000000 P 62546b3a10ed4f62818cf3016b3a0d9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31cb951a80fc476f97c0a9298b27f27e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34529 } } peers { permanent_uuid: "bcbfd113b70c43bf986f91ee121fa456" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37441 } } peers { permanent_uuid: "62546b3a10ed4f62818cf3016b3a0d9c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37985 } }
I20260620 04:27:31.771478 13760 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 31cb951a80fc476f97c0a9298b27f27e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62546b3a10ed4f62818cf3016b3a0d9c in term 1.
I20260620 04:27:31.772152 13954 sys_catalog.cc:455] T 00000000000000000000000000000000 P bcbfd113b70c43bf986f91ee121fa456 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31cb951a80fc476f97c0a9298b27f27e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34529 } } peers { permanent_uuid: "bcbfd113b70c43bf986f91ee121fa456" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37441 } } peers { permanent_uuid: "62546b3a10ed4f62818cf3016b3a0d9c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37985 } } }
I20260620 04:27:31.772286 13954 sys_catalog.cc:458] T 00000000000000000000000000000000 P bcbfd113b70c43bf986f91ee121fa456 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 04:27:31.760280 13718 master_runner.cc:186] Error getting master registration for 127.0.98.61:37441: OK, instance_id { permanent_uuid: "bcbfd113b70c43bf986f91ee121fa456" instance_seqno: 1781929651471538 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 04:27:31.773440 13718 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 04:27:31.773964 13961 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62546b3a10ed4f62818cf3016b3a0d9c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 62546b3a10ed4f62818cf3016b3a0d9c. Latest consensus state: current_term: 1 leader_uuid: "62546b3a10ed4f62818cf3016b3a0d9c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31cb951a80fc476f97c0a9298b27f27e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34529 } } peers { permanent_uuid: "bcbfd113b70c43bf986f91ee121fa456" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37441 } } peers { permanent_uuid: "62546b3a10ed4f62818cf3016b3a0d9c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37985 } } }
I20260620 04:27:31.774113 13961 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62546b3a10ed4f62818cf3016b3a0d9c [sys.catalog]: This master's current role is: LEADER
I20260620 04:27:31.774677 13867 master_runner.cc:232] Existing masters have differing terms: 0 vs 1
I20260620 04:27:31.774775 13867 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 04:27:31.775928 13793 master_runner.cc:232] Existing masters have differing terms: 0 vs 1
I20260620 04:27:31.776062 13793 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 04:27:31.776245 13952 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31cb951a80fc476f97c0a9298b27f27e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31cb951a80fc476f97c0a9298b27f27e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34529 } } peers { permanent_uuid: "bcbfd113b70c43bf986f91ee121fa456" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37441 } } peers { permanent_uuid: "62546b3a10ed4f62818cf3016b3a0d9c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37985 } } }
I20260620 04:27:31.776356 13952 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31cb951a80fc476f97c0a9298b27f27e [sys.catalog]: This master's current role is: FOLLOWER
W20260620 04:27:31.777794 13983 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 62546b3a10ed4f62818cf3016b3a0d9c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 04:27:31.777875 13983 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 04:27:31.777931 13992 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:27:31.779299 13992 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260620 04:27:31.780251 13989 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bcbfd113b70c43bf986f91ee121fa456: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 04:27:31.780318 13989 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 04:27:31.785086 13760 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 31cb951a80fc476f97c0a9298b27f27e [term 1 FOLLOWER]: Refusing update from remote peer 62546b3a10ed4f62818cf3016b3a0d9c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:27:31.785749 13953 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 62546b3a10ed4f62818cf3016b3a0d9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "31cb951a80fc476f97c0a9298b27f27e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:27:31.786746 13835 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bcbfd113b70c43bf986f91ee121fa456 [term 1 FOLLOWER]: Refusing update from remote peer 62546b3a10ed4f62818cf3016b3a0d9c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:27:31.787266 13953 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 62546b3a10ed4f62818cf3016b3a0d9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcbfd113b70c43bf986f91ee121fa456" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:27:31.790017 13952 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31cb951a80fc476f97c0a9298b27f27e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 62546b3a10ed4f62818cf3016b3a0d9c. Latest consensus state: current_term: 1 leader_uuid: "62546b3a10ed4f62818cf3016b3a0d9c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31cb951a80fc476f97c0a9298b27f27e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34529 } } peers { permanent_uuid: "bcbfd113b70c43bf986f91ee121fa456" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37441 } } peers { permanent_uuid: "62546b3a10ed4f62818cf3016b3a0d9c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37985 } } }
I20260620 04:27:31.790148 13952 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31cb951a80fc476f97c0a9298b27f27e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 04:27:31.793622 13954 sys_catalog.cc:455] T 00000000000000000000000000000000 P bcbfd113b70c43bf986f91ee121fa456 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 62546b3a10ed4f62818cf3016b3a0d9c. Latest consensus state: current_term: 1 leader_uuid: "62546b3a10ed4f62818cf3016b3a0d9c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31cb951a80fc476f97c0a9298b27f27e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34529 } } peers { permanent_uuid: "bcbfd113b70c43bf986f91ee121fa456" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37441 } } peers { permanent_uuid: "62546b3a10ed4f62818cf3016b3a0d9c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37985 } } }
I20260620 04:27:31.793751 13954 sys_catalog.cc:458] T 00000000000000000000000000000000 P bcbfd113b70c43bf986f91ee121fa456 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 04:27:31.795001 13953 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62546b3a10ed4f62818cf3016b3a0d9c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "62546b3a10ed4f62818cf3016b3a0d9c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31cb951a80fc476f97c0a9298b27f27e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34529 } } peers { permanent_uuid: "bcbfd113b70c43bf986f91ee121fa456" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37441 } } peers { permanent_uuid: "62546b3a10ed4f62818cf3016b3a0d9c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37985 } } }
I20260620 04:27:31.795117 13953 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62546b3a10ed4f62818cf3016b3a0d9c [sys.catalog]: This master's current role is: LEADER
I20260620 04:27:31.795229 13995 mvcc.cc:205] Tried to move back new op lower bound from 7298783853702799360 to 7298783853656031232. Current Snapshot: MvccSnapshot[applied={T|T < 7298783853702799360}]
I20260620 04:27:31.797166 13952 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31cb951a80fc476f97c0a9298b27f27e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "62546b3a10ed4f62818cf3016b3a0d9c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31cb951a80fc476f97c0a9298b27f27e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34529 } } peers { permanent_uuid: "bcbfd113b70c43bf986f91ee121fa456" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37441 } } peers { permanent_uuid: "62546b3a10ed4f62818cf3016b3a0d9c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37985 } } }
I20260620 04:27:31.797287 13952 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31cb951a80fc476f97c0a9298b27f27e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 04:27:31.798622 13961 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62546b3a10ed4f62818cf3016b3a0d9c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "62546b3a10ed4f62818cf3016b3a0d9c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31cb951a80fc476f97c0a9298b27f27e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34529 } } peers { permanent_uuid: "bcbfd113b70c43bf986f91ee121fa456" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37441 } } peers { permanent_uuid: "62546b3a10ed4f62818cf3016b3a0d9c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37985 } } }
I20260620 04:27:31.798738 13961 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62546b3a10ed4f62818cf3016b3a0d9c [sys.catalog]: This master's current role is: LEADER
I20260620 04:27:31.799885 13954 sys_catalog.cc:455] T 00000000000000000000000000000000 P bcbfd113b70c43bf986f91ee121fa456 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "62546b3a10ed4f62818cf3016b3a0d9c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31cb951a80fc476f97c0a9298b27f27e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34529 } } peers { permanent_uuid: "bcbfd113b70c43bf986f91ee121fa456" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37441 } } peers { permanent_uuid: "62546b3a10ed4f62818cf3016b3a0d9c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37985 } } }
I20260620 04:27:31.800019 13954 sys_catalog.cc:458] T 00000000000000000000000000000000 P bcbfd113b70c43bf986f91ee121fa456 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 04:27:31.801470 13998 mvcc.cc:205] Tried to move back new op lower bound from 7298783853702799360 to 7298783853656031232. Current Snapshot: MvccSnapshot[applied={T|T < 7298783853702799360}]
I20260620 04:27:31.806326 13992 catalog_manager.cc:1347] Generated new cluster ID: 41024366af124355a2a9486de86eff15
I20260620 04:27:31.806423 13992 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:27:31.821694 13992 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:27:31.823127 13992 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:27:31.839000 13992 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 62546b3a10ed4f62818cf3016b3a0d9c: Generated new TSK 0
I20260620 04:27:31.840013 13992 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 04:27:32.771234 13977 catalog_manager.cc:1259] Loaded cluster ID: 41024366af124355a2a9486de86eff15
I20260620 04:27:32.771311 13977 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 31cb951a80fc476f97c0a9298b27f27e: loading cluster ID for follower catalog manager: success
I20260620 04:27:32.773047 13977 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 31cb951a80fc476f97c0a9298b27f27e: acquiring CA information for follower catalog manager: success
I20260620 04:27:32.774188 13977 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 31cb951a80fc476f97c0a9298b27f27e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 04:27:32.781919 13989 catalog_manager.cc:1259] Loaded cluster ID: 41024366af124355a2a9486de86eff15
I20260620 04:27:32.781993 13989 catalog_manager.cc:1552] T 00000000000000000000000000000000 P bcbfd113b70c43bf986f91ee121fa456: loading cluster ID for follower catalog manager: success
I20260620 04:27:32.784368 13989 catalog_manager.cc:1574] T 00000000000000000000000000000000 P bcbfd113b70c43bf986f91ee121fa456: acquiring CA information for follower catalog manager: success
I20260620 04:27:32.786094 13989 catalog_manager.cc:1602] T 00000000000000000000000000000000 P bcbfd113b70c43bf986f91ee121fa456: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 04:27:32.914156 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 13718
I20260620 04:27:32.919647 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 13793
I20260620 04:27:32.924423 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 13867
2026-06-20T04:27:32Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2882 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781929496330398-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:27:32 dist-test-slave-nk6d krb5kdc[14010](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:27:32 dist-test-slave-nk6d krb5kdc[14010](info): set up 2 sockets
Jun 20 04:27:32 dist-test-slave-nk6d krb5kdc[14010](info): commencing operation
krb5kdc: starting...
W20260620 04:27:33.872033 392 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781929496330398-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781929496330398-392-0/krb5kdc/test-user.keytab.
Jun 20 04:27:33 dist-test-slave-nk6d krb5kdc[14010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929653, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:27:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:27:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 04:27:33.982681 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781929496330398-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781929496330398-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781929496330398-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33269
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33269
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:27:34.090067 14023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:27:34.090353 14023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:27:34.090408 14023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:27:34.094429 14023 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:27:34.094511 14023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:27:34.094543 14023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:27:34.094565 14023 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:27:34.094586 14023 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:27:34.099977 14023 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781929496330398-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781929496330398-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33269
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33269
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14023
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781929496330398-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:27:34.101048 14023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:27:34.102043 14023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:27:34.110224 14031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:27:34.110234 14029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:27:34.110240 14028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:27:34.110525 14023 server_base.cc:1034] running on GCE node
I20260620 04:27:34.111254 14023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:27:34.112491 14023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:27:34.113672 14023 hybrid_clock.cc:648] HybridClock initialized: now 1781929654113644 us; error 37 us; skew 500 ppm
Jun 20 04:27:34 dist-test-slave-nk6d krb5kdc[14010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929654, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:27:34.116397 14023 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 04:27:34.117308 14023 webserver.cc:460] Webserver started at http://127.0.98.62:41555/ using document root <none> and password file <none>
I20260620 04:27:34.117848 14023 fs_manager.cc:359] Metadata directory not provided
I20260620 04:27:34.117914 14023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:27:34.118099 14023 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:27:34.120162 14023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781929496330398-392-0/minicluster-data/master-0/data/instance:
uuid: "fe71bf8affea475fab5c6b26143e9479"
format_stamp: "Formatted at 2026-06-20 04:27:34 on dist-test-slave-nk6d"
server_key: "c78f7af631f2dd51c577babbe4f56563"
server_key_iv: "0c4f8fafdf3c36fca9430e0b75a15e30"
server_key_version: "encryptionkey@0"
I20260620 04:27:34.120700 14023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781929496330398-392-0/minicluster-data/master-0/wal/instance:
uuid: "fe71bf8affea475fab5c6b26143e9479"
format_stamp: "Formatted at 2026-06-20 04:27:34 on dist-test-slave-nk6d"
server_key: "c78f7af631f2dd51c577babbe4f56563"
server_key_iv: "0c4f8fafdf3c36fca9430e0b75a15e30"
server_key_version: "encryptionkey@0"
I20260620 04:27:34.124325 14023 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:27:34.126657 14038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:27:34.127892 14023 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:27:34.128041 14023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781929496330398-392-0/minicluster-data/master-0/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781929496330398-392-0/minicluster-data/master-0/wal
uuid: "fe71bf8affea475fab5c6b26143e9479"
format_stamp: "Formatted at 2026-06-20 04:27:34 on dist-test-slave-nk6d"
server_key: "c78f7af631f2dd51c577babbe4f56563"
server_key_iv: "0c4f8fafdf3c36fca9430e0b75a15e30"
server_key_version: "encryptionkey@0"
I20260620 04:27:34.128134 14023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781929496330398-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781929496330398-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781929496330398-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:27:34.140390 14023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:27:34.143352 14023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:27:34.143565 14023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:27:34.150395 14023 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33269
I20260620 04:27:34.150446 14090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33269 every 8 connection(s)
I20260620 04:27:34.151336 14023 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:27:34.154012 14091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:27:34.158146 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 14023
I20260620 04:27:34.158257 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781929496330398-392-0/minicluster-data/master-0/wal/instance
I20260620 04:27:34.158528 392 external_mini_cluster.cc:1428] Setting key eda550dc1bd8f77bef5d9091cedf4f49
I20260620 04:27:34.159680 14091 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe71bf8affea475fab5c6b26143e9479: Bootstrap starting.
I20260620 04:27:34.161955 14091 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fe71bf8affea475fab5c6b26143e9479: Neither blocks nor log segments found. Creating new log.
I20260620 04:27:34.162809 14091 log.cc:824] T 00000000000000000000000000000000 P fe71bf8affea475fab5c6b26143e9479: Log is configured to *not* fsync() on all Append() calls
I20260620 04:27:34.164794 14091 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe71bf8affea475fab5c6b26143e9479: No bootstrap required, opened a new log
Jun 20 04:27:34 dist-test-slave-nk6d krb5kdc[14010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929653, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:27:34.168890 14091 raft_consensus.cc:348] T 00000000000000000000000000000000 P fe71bf8affea475fab5c6b26143e9479 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe71bf8affea475fab5c6b26143e9479" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33269 } }
I20260620 04:27:34.169176 14091 raft_consensus.cc:374] T 00000000000000000000000000000000 P fe71bf8affea475fab5c6b26143e9479 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:27:34.169241 14091 raft_consensus.cc:729] T 00000000000000000000000000000000 P fe71bf8affea475fab5c6b26143e9479 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe71bf8affea475fab5c6b26143e9479, State: Initialized, Role: FOLLOWER
I20260620 04:27:34.169713 14091 consensus_queue.cc:260] T 00000000000000000000000000000000 P fe71bf8affea475fab5c6b26143e9479 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe71bf8affea475fab5c6b26143e9479" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33269 } }
I20260620 04:27:34.169864 14091 raft_consensus.cc:388] T 00000000000000000000000000000000 P fe71bf8affea475fab5c6b26143e9479 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:27:34.169929 14091 raft_consensus.cc:482] T 00000000000000000000000000000000 P fe71bf8affea475fab5c6b26143e9479 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:27:34.170032 14091 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fe71bf8affea475fab5c6b26143e9479 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:27:34.171031 14091 raft_consensus.cc:504] T 00000000000000000000000000000000 P fe71bf8affea475fab5c6b26143e9479 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe71bf8affea475fab5c6b26143e9479" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33269 } }
I20260620 04:27:34.171379 14091 leader_election.cc:302] T 00000000000000000000000000000000 P fe71bf8affea475fab5c6b26143e9479 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fe71bf8affea475fab5c6b26143e9479; no voters:
I20260620 04:27:34.171723 14091 leader_election.cc:288] T 00000000000000000000000000000000 P fe71bf8affea475fab5c6b26143e9479 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:27:34.171926 14096 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fe71bf8affea475fab5c6b26143e9479 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:27:34.172178 14096 raft_consensus.cc:686] T 00000000000000000000000000000000 P fe71bf8affea475fab5c6b26143e9479 [term 1 LEADER]: Becoming Leader. State: Replica: fe71bf8affea475fab5c6b26143e9479, State: Running, Role: LEADER
I20260620 04:27:34.172470 14096 consensus_queue.cc:237] T 00000000000000000000000000000000 P fe71bf8affea475fab5c6b26143e9479 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe71bf8affea475fab5c6b26143e9479" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33269 } }
I20260620 04:27:34.172845 14094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:34.159779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54024 (local address 127.0.98.62:33269)
0620 04:27:34.160253 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0620 04:27:34.160269 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 04:27:34.160306 (+ 37us) server_negotiation.cc:407] Connection header received
0620 04:27:34.161029 (+ 723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:27:34.161046 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:27:34.161385 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:27:34.161687 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:27:34.162615 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:34.163637 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:34.164776 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:34.165279 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:34.165400 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:27:34.167911 (+ 2511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:27:34.167917 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:27:34.167934 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:27:34.167996 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:27:34.169772 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:27:34.170210 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:27:34.170216 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:27:34.170222 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:27:34.170350 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:27:34.170624 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:27:34.170627 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:27:34.170629 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:27:34.171122 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:27:34.171335 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0620 04:27:34.171599 (+ 264us) server_negotiation.cc:299] Negotiation successful
0620 04:27:34.171886 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":171,"threads_started":1}
I20260620 04:27:34.172942 14091 sys_catalog.cc:564] T 00000000000000000000000000000000 P fe71bf8affea475fab5c6b26143e9479 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:27:34.173537 14096 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe71bf8affea475fab5c6b26143e9479 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fe71bf8affea475fab5c6b26143e9479. Latest consensus state: current_term: 1 leader_uuid: "fe71bf8affea475fab5c6b26143e9479" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe71bf8affea475fab5c6b26143e9479" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33269 } } }
I20260620 04:27:34.173642 14096 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe71bf8affea475fab5c6b26143e9479 [sys.catalog]: This master's current role is: LEADER
I20260620 04:27:34.173903 14097 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe71bf8affea475fab5c6b26143e9479 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fe71bf8affea475fab5c6b26143e9479" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe71bf8affea475fab5c6b26143e9479" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33269 } } }
I20260620 04:27:34.174003 14097 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe71bf8affea475fab5c6b26143e9479 [sys.catalog]: This master's current role is: LEADER
I20260620 04:27:34.174099 14103 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:27:34.175977 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 04:27:34.177270 14103 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:27:34.183037 14103 catalog_manager.cc:1347] Generated new cluster ID: c82c3e33aac8421588cbcf9c9768177d
I20260620 04:27:34.183095 14103 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:27:34.205679 14103 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:27:34.206665 14103 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:27:34.214114 14103 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fe71bf8affea475fab5c6b26143e9479: Generated new TSK 0
I20260620 04:27:34.214804 14103 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 04:27:34.231655 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 14023
2026-06-20T04:27:34Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1306 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:27:34 dist-test-slave-nk6d krb5kdc[14118](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:27:34 dist-test-slave-nk6d krb5kdc[14118](info): set up 2 sockets
Jun 20 04:27:34 dist-test-slave-nk6d krb5kdc[14118](info): commencing operation
krb5kdc: starting...
W20260620 04:27:35.172137 392 mini_kdc.cc:121] Time spent starting KDC: real 0.922s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/krb5kdc/test-user.keytab.
Jun 20 04:27:35 dist-test-slave-nk6d krb5kdc[14118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929655, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:27:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:27:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 04:27:35.282027 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36915
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36915
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:27:35.387302 14131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:27:35.387563 14131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:27:35.387605 14131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:27:35.391525 14131 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:27:35.391575 14131 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:27:35.391593 14131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:27:35.391611 14131 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:27:35.391639 14131 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:27:35.396804 14131 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36915
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36915
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14131
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:27:35.397773 14131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:27:35.398770 14131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:27:35.406770 14139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:27:35.406796 14137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:27:35.406821 14136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:27:35.407029 14131 server_base.cc:1034] running on GCE node
I20260620 04:27:35.407660 14131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:27:35.408799 14131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:27:35.409983 14131 hybrid_clock.cc:648] HybridClock initialized: now 1781929655409960 us; error 33 us; skew 500 ppm
Jun 20 04:27:35 dist-test-slave-nk6d krb5kdc[14118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929655, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:27:35.412781 14131 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 04:27:35.413615 14131 webserver.cc:460] Webserver started at http://127.0.98.62:35683/ using document root <none> and password file <none>
I20260620 04:27:35.414149 14131 fs_manager.cc:359] Metadata directory not provided
I20260620 04:27:35.414215 14131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:27:35.414424 14131 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:27:35.416452 14131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/master-0/data/instance:
uuid: "a87a20f134ed4ae9a447a40942f3d7ea"
format_stamp: "Formatted at 2026-06-20 04:27:35 on dist-test-slave-nk6d"
server_key: "812718ef149fdf619ec957fc50632200"
server_key_iv: "1502d5d4f62daadbaabd2bdeee1d2496"
server_key_version: "encryptionkey@0"
I20260620 04:27:35.416982 14131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/master-0/wal/instance:
uuid: "a87a20f134ed4ae9a447a40942f3d7ea"
format_stamp: "Formatted at 2026-06-20 04:27:35 on dist-test-slave-nk6d"
server_key: "812718ef149fdf619ec957fc50632200"
server_key_iv: "1502d5d4f62daadbaabd2bdeee1d2496"
server_key_version: "encryptionkey@0"
I20260620 04:27:35.420533 14131 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 04:27:35.422937 14146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:27:35.424233 14131 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:27:35.424345 14131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/master-0/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/master-0/wal
uuid: "a87a20f134ed4ae9a447a40942f3d7ea"
format_stamp: "Formatted at 2026-06-20 04:27:35 on dist-test-slave-nk6d"
server_key: "812718ef149fdf619ec957fc50632200"
server_key_iv: "1502d5d4f62daadbaabd2bdeee1d2496"
server_key_version: "encryptionkey@0"
I20260620 04:27:35.424437 14131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:27:35.439002 14131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:27:35.441848 14131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:27:35.442015 14131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:27:35.448321 14131 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36915
I20260620 04:27:35.448546 14198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36915 every 8 connection(s)
I20260620 04:27:35.449249 14131 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:27:35.451897 14199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:27:35.457146 14199 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a87a20f134ed4ae9a447a40942f3d7ea: Bootstrap starting.
I20260620 04:27:35.457635 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 14131
I20260620 04:27:35.457772 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/master-0/wal/instance
I20260620 04:27:35.458061 392 external_mini_cluster.cc:1428] Setting key ab0d32c53eb5f54bb4e37dd67a49082a
I20260620 04:27:35.459759 14199 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a87a20f134ed4ae9a447a40942f3d7ea: Neither blocks nor log segments found. Creating new log.
I20260620 04:27:35.460471 14199 log.cc:824] T 00000000000000000000000000000000 P a87a20f134ed4ae9a447a40942f3d7ea: Log is configured to *not* fsync() on all Append() calls
I20260620 04:27:35.462499 14199 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a87a20f134ed4ae9a447a40942f3d7ea: No bootstrap required, opened a new log
I20260620 04:27:35.466352 14199 raft_consensus.cc:348] T 00000000000000000000000000000000 P a87a20f134ed4ae9a447a40942f3d7ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a87a20f134ed4ae9a447a40942f3d7ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36915 } }
I20260620 04:27:35.466568 14199 raft_consensus.cc:374] T 00000000000000000000000000000000 P a87a20f134ed4ae9a447a40942f3d7ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:27:35.466631 14199 raft_consensus.cc:729] T 00000000000000000000000000000000 P a87a20f134ed4ae9a447a40942f3d7ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a87a20f134ed4ae9a447a40942f3d7ea, State: Initialized, Role: FOLLOWER
Jun 20 04:27:35 dist-test-slave-nk6d krb5kdc[14118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929655, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:27:35.467063 14199 consensus_queue.cc:260] T 00000000000000000000000000000000 P a87a20f134ed4ae9a447a40942f3d7ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a87a20f134ed4ae9a447a40942f3d7ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36915 } }
I20260620 04:27:35.467193 14199 raft_consensus.cc:388] T 00000000000000000000000000000000 P a87a20f134ed4ae9a447a40942f3d7ea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:27:35.467262 14199 raft_consensus.cc:482] T 00000000000000000000000000000000 P a87a20f134ed4ae9a447a40942f3d7ea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:27:35.467368 14199 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a87a20f134ed4ae9a447a40942f3d7ea [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:27:35.468494 14199 raft_consensus.cc:504] T 00000000000000000000000000000000 P a87a20f134ed4ae9a447a40942f3d7ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a87a20f134ed4ae9a447a40942f3d7ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36915 } }
I20260620 04:27:35.468858 14199 leader_election.cc:302] T 00000000000000000000000000000000 P a87a20f134ed4ae9a447a40942f3d7ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a87a20f134ed4ae9a447a40942f3d7ea; no voters:
I20260620 04:27:35.469187 14199 leader_election.cc:288] T 00000000000000000000000000000000 P a87a20f134ed4ae9a447a40942f3d7ea [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:27:35.469311 14204 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a87a20f134ed4ae9a447a40942f3d7ea [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:27:35.469511 14204 raft_consensus.cc:686] T 00000000000000000000000000000000 P a87a20f134ed4ae9a447a40942f3d7ea [term 1 LEADER]: Becoming Leader. State: Replica: a87a20f134ed4ae9a447a40942f3d7ea, State: Running, Role: LEADER
I20260620 04:27:35.469859 14204 consensus_queue.cc:237] T 00000000000000000000000000000000 P a87a20f134ed4ae9a447a40942f3d7ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a87a20f134ed4ae9a447a40942f3d7ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36915 } }
I20260620 04:27:35.470469 14199 sys_catalog.cc:564] T 00000000000000000000000000000000 P a87a20f134ed4ae9a447a40942f3d7ea [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:27:35.471803 14206 sys_catalog.cc:455] T 00000000000000000000000000000000 P a87a20f134ed4ae9a447a40942f3d7ea [sys.catalog]: SysCatalogTable state changed. Reason: New leader a87a20f134ed4ae9a447a40942f3d7ea. Latest consensus state: current_term: 1 leader_uuid: "a87a20f134ed4ae9a447a40942f3d7ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a87a20f134ed4ae9a447a40942f3d7ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36915 } } }
I20260620 04:27:35.471912 14206 sys_catalog.cc:458] T 00000000000000000000000000000000 P a87a20f134ed4ae9a447a40942f3d7ea [sys.catalog]: This master's current role is: LEADER
I20260620 04:27:35.472097 14205 sys_catalog.cc:455] T 00000000000000000000000000000000 P a87a20f134ed4ae9a447a40942f3d7ea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a87a20f134ed4ae9a447a40942f3d7ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a87a20f134ed4ae9a447a40942f3d7ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36915 } } }
I20260620 04:27:35.472178 14205 sys_catalog.cc:458] T 00000000000000000000000000000000 P a87a20f134ed4ae9a447a40942f3d7ea [sys.catalog]: This master's current role is: LEADER
I20260620 04:27:35.472544 14202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:35.459302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42242 (local address 127.0.98.62:36915)
0620 04:27:35.459678 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0620 04:27:35.459693 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 04:27:35.459727 (+ 34us) server_negotiation.cc:407] Connection header received
0620 04:27:35.460425 (+ 698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:27:35.460443 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:27:35.460798 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:27:35.461069 (+ 271us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:27:35.461772 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:35.463188 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:35.464115 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:35.464817 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:35.464981 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:27:35.467330 (+ 2349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:27:35.467337 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:27:35.467353 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:27:35.467386 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:27:35.469281 (+ 1895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:27:35.469748 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:27:35.469754 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:27:35.469760 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:27:35.469898 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:27:35.470181 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:27:35.470184 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:27:35.470186 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:27:35.470702 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:27:35.470902 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0620 04:27:35.471156 (+ 254us) server_negotiation.cc:299] Negotiation successful
0620 04:27:35.471476 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":102,"threads_started":1}
W20260620 04:27:35.475194 14219 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a87a20f134ed4ae9a447a40942f3d7ea: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 04:27:35.475252 14219 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 04:27:35.475438 14218 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:27:35.476382 14218 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:27:35.481652 14218 catalog_manager.cc:1347] Generated new cluster ID: 676c7b67d90c4e8a855460a0f74a3aa4
I20260620 04:27:35.481710 14218 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:27:35.494925 14218 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:27:35.496788 14218 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:27:35.506961 14218 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a87a20f134ed4ae9a447a40942f3d7ea: Generated new TSK 0
I20260620 04:27:35.507786 14218 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 04:27:35.536682 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36915
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:27:35.644734 14225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:27:35.644981 14225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:27:35.645023 14225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:27:35.648958 14225 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:27:35.649011 14225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:27:35.649066 14225 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 04:27:35.654129 14225 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36915
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14225
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:27:35.655102 14225 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:27:35.656097 14225 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:27:35.665097 14230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:27:35.665097 14233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:27:35.665098 14231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:27:35.665354 14225 server_base.cc:1034] running on GCE node
I20260620 04:27:35.665746 14225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:27:35.666327 14225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:27:35.667480 14225 hybrid_clock.cc:648] HybridClock initialized: now 1781929655667461 us; error 30 us; skew 500 ppm
Jun 20 04:27:35 dist-test-slave-nk6d krb5kdc[14118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929655, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:27:35.670095 14225 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 04:27:35.670938 14225 webserver.cc:460] Webserver started at http://127.0.98.1:36093/ using document root <none> and password file <none>
I20260620 04:27:35.671446 14225 fs_manager.cc:359] Metadata directory not provided
I20260620 04:27:35.671499 14225 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:27:35.671700 14225 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:27:35.673688 14225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-0/data/instance:
uuid: "a0a07d8cbfd141e49fcce98f279a8ebc"
format_stamp: "Formatted at 2026-06-20 04:27:35 on dist-test-slave-nk6d"
server_key: "a118e53c149d76aa58ea1913d93a045b"
server_key_iv: "536eba379b58ca4676c768769cb42be3"
server_key_version: "encryptionkey@0"
I20260620 04:27:35.674218 14225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-0/wal/instance:
uuid: "a0a07d8cbfd141e49fcce98f279a8ebc"
format_stamp: "Formatted at 2026-06-20 04:27:35 on dist-test-slave-nk6d"
server_key: "a118e53c149d76aa58ea1913d93a045b"
server_key_iv: "536eba379b58ca4676c768769cb42be3"
server_key_version: "encryptionkey@0"
I20260620 04:27:35.677747 14225 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 04:27:35.680179 14240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:27:35.681457 14225 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:27:35.681567 14225 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-0/wal
uuid: "a0a07d8cbfd141e49fcce98f279a8ebc"
format_stamp: "Formatted at 2026-06-20 04:27:35 on dist-test-slave-nk6d"
server_key: "a118e53c149d76aa58ea1913d93a045b"
server_key_iv: "536eba379b58ca4676c768769cb42be3"
server_key_version: "encryptionkey@0"
I20260620 04:27:35.681664 14225 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:27:35.695042 14225 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:27:35.698194 14225 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:27:35.698376 14225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:27:35.698936 14225 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:27:35.699734 14225 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:27:35.699779 14225 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:27:35.699828 14225 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:27:35.699856 14225 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:27:35.708083 14225 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36255
I20260620 04:27:35.708108 14353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36255 every 8 connection(s)
I20260620 04:27:35.709021 14225 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-0/data/info.pb
I20260620 04:27:35.712404 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 14225
I20260620 04:27:35.712522 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-0/wal/instance
I20260620 04:27:35.712798 392 external_mini_cluster.cc:1428] Setting key 8b32cf163eb75c8072c03339f3102e71
Jun 20 04:27:35 dist-test-slave-nk6d krb5kdc[14118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929655, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 04:27:35.722566 14202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:35.710724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38175 (local address 127.0.98.62:36915)
0620 04:27:35.710850 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 04:27:35.710854 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:27:35.711727 (+ 873us) server_negotiation.cc:407] Connection header received
0620 04:27:35.713087 (+ 1360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:27:35.713090 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:27:35.713149 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:27:35.713225 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:27:35.714380 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:35.714911 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:35.716414 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:35.716695 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:35.716787 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:27:35.719101 (+ 2314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:27:35.719104 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:27:35.719107 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:27:35.719132 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:27:35.720548 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:27:35.721098 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:27:35.721102 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:27:35.721103 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:27:35.721180 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:27:35.721506 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:27:35.721509 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:27:35.721511 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:27:35.721724 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:27:35.721806 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 04:27:35.722305 (+ 499us) server_negotiation.cc:299] Negotiation successful
0620 04:27:35.722440 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 04:27:35.723402 14356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:35.711062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36915 (local address 127.0.98.1:38175)
0620 04:27:35.711558 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:35.711600 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 04:27:35.712876 (+ 1276us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:27:35.713407 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:27:35.713414 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:27:35.713762 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:27:35.714238 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:27:35.714252 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:27:35.715028 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:27:35.715031 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:27:35.716297 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:27:35.716309 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:27:35.716843 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:27:35.716846 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:27:35.717003 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:27:35.717546 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:27:35.717563 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:27:35.718943 (+ 1380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:27:35.720657 (+ 1714us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:27:35.720663 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:27:35.720678 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:27:35.721005 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:27:35.721261 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:27:35.721265 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:27:35.721267 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:27:35.721405 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:27:35.721813 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:27:35.721820 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:27:35.722054 (+ 234us) client_negotiation.cc:769] Sending connection context
0620 04:27:35.722204 (+ 150us) client_negotiation.cc:240] Negotiation successful
0620 04:27:35.722540 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":136,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 04:27:35.724694 14354 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36915
I20260620 04:27:35.724916 14354 heartbeater.cc:464] Registering TS with master...
I20260620 04:27:35.725479 14354 heartbeater.cc:511] Master 127.0.98.62:36915 requested a full tablet report, sending...
I20260620 04:27:35.726999 14163 ts_manager.cc:194] Registered new tserver with Master: a0a07d8cbfd141e49fcce98f279a8ebc (127.0.98.1:36255)
I20260620 04:27:35.728178 14163 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:38175
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 04:27:35.736879 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36915
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:27:35.845932 14359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:27:35.846202 14359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:27:35.846246 14359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:27:35.850354 14359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:27:35.850417 14359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:27:35.850473 14359 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 04:27:35.855537 14359 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36915
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14359
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:27:35.856509 14359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:27:35.857450 14359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:27:35.866210 14364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:27:35.866204 14367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:27:35.866225 14365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:27:35.866394 14359 server_base.cc:1034] running on GCE node
I20260620 04:27:35.866816 14359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:27:35.867419 14359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:27:35.868556 14359 hybrid_clock.cc:648] HybridClock initialized: now 1781929655868529 us; error 39 us; skew 500 ppm
Jun 20 04:27:35 dist-test-slave-nk6d krb5kdc[14118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929655, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:27:35.871160 14359 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 04:27:35.872004 14359 webserver.cc:460] Webserver started at http://127.0.98.2:43001/ using document root <none> and password file <none>
I20260620 04:27:35.872514 14359 fs_manager.cc:359] Metadata directory not provided
I20260620 04:27:35.872570 14359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:27:35.872721 14359 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:27:35.874603 14359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-1/data/instance:
uuid: "126db552d7e645d597ecbab7867c8368"
format_stamp: "Formatted at 2026-06-20 04:27:35 on dist-test-slave-nk6d"
server_key: "41c6a4829c42a7f128ddc2edb2c9fa7f"
server_key_iv: "ad041497cafe3adab320c0e4650a0859"
server_key_version: "encryptionkey@0"
I20260620 04:27:35.875097 14359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-1/wal/instance:
uuid: "126db552d7e645d597ecbab7867c8368"
format_stamp: "Formatted at 2026-06-20 04:27:35 on dist-test-slave-nk6d"
server_key: "41c6a4829c42a7f128ddc2edb2c9fa7f"
server_key_iv: "ad041497cafe3adab320c0e4650a0859"
server_key_version: "encryptionkey@0"
I20260620 04:27:35.878587 14359 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 04:27:35.880848 14374 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:27:35.882078 14359 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:27:35.882180 14359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-1/wal
uuid: "126db552d7e645d597ecbab7867c8368"
format_stamp: "Formatted at 2026-06-20 04:27:35 on dist-test-slave-nk6d"
server_key: "41c6a4829c42a7f128ddc2edb2c9fa7f"
server_key_iv: "ad041497cafe3adab320c0e4650a0859"
server_key_version: "encryptionkey@0"
I20260620 04:27:35.882292 14359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:27:35.894865 14359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:27:35.897678 14359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:27:35.897850 14359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:27:35.898408 14359 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:27:35.899189 14359 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:27:35.899243 14359 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:27:35.899296 14359 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:27:35.899323 14359 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:27:35.907577 14359 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43567
I20260620 04:27:35.907606 14487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43567 every 8 connection(s)
I20260620 04:27:35.908453 14359 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-1/data/info.pb
I20260620 04:27:35.912489 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 14359
I20260620 04:27:35.912623 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-1/wal/instance
I20260620 04:27:35.912920 392 external_mini_cluster.cc:1428] Setting key 6bec8ea8b6688ddb02f7e8c798e3d055
Jun 20 04:27:35 dist-test-slave-nk6d krb5kdc[14118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929655, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 04:27:35.922135 14202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:35.910149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48609 (local address 127.0.98.62:36915)
0620 04:27:35.910288 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 04:27:35.910292 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:27:35.911324 (+ 1032us) server_negotiation.cc:407] Connection header received
0620 04:27:35.912273 (+ 949us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:27:35.912276 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:27:35.912334 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:27:35.912408 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:27:35.913790 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:35.914349 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:35.915844 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:35.916110 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:35.916214 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:27:35.918694 (+ 2480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:27:35.918697 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:27:35.918700 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:27:35.918727 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:27:35.920169 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:27:35.920663 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:27:35.920667 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:27:35.920669 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:27:35.920742 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:27:35.921044 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:27:35.921047 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:27:35.921048 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:27:35.921256 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:27:35.921341 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 04:27:35.921845 (+ 504us) server_negotiation.cc:299] Negotiation successful
0620 04:27:35.921968 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":17,"server-negotiator.queue_time_us":45}
I20260620 04:27:35.923103 14490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:35.910519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36915 (local address 127.0.98.2:48609)
0620 04:27:35.911167 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:35.911207 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 04:27:35.912063 (+ 856us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:27:35.912601 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:27:35.912610 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:27:35.913040 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:27:35.913635 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:27:35.913660 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:27:35.914512 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:27:35.914515 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:27:35.915709 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:27:35.915721 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:27:35.916259 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:27:35.916263 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:27:35.916433 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:27:35.916995 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:27:35.917013 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:27:35.918512 (+ 1499us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:27:35.920284 (+ 1772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:27:35.920290 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:27:35.920308 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:27:35.920570 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:27:35.920820 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:27:35.920823 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:27:35.920825 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:27:35.920959 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:27:35.921358 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:27:35.921365 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:27:35.921614 (+ 249us) client_negotiation.cc:769] Sending connection context
0620 04:27:35.921816 (+ 202us) client_negotiation.cc:240] Negotiation successful
0620 04:27:35.922153 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":119,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260620 04:27:35.924361 14488 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36915
I20260620 04:27:35.924597 14488 heartbeater.cc:464] Registering TS with master...
I20260620 04:27:35.925072 14488 heartbeater.cc:511] Master 127.0.98.62:36915 requested a full tablet report, sending...
I20260620 04:27:35.926106 14163 ts_manager.cc:194] Registered new tserver with Master: 126db552d7e645d597ecbab7867c8368 (127.0.98.2:43567)
I20260620 04:27:35.926654 14163 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:48609
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 04:27:35.936465 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36915
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:27:36.045243 14493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:27:36.045553 14493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:27:36.045625 14493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:27:36.049787 14493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:27:36.049866 14493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:27:36.049960 14493 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 04:27:36.055657 14493 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36915
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14493
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:27:36.056797 14493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:27:36.057838 14493 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:27:36.067176 14498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:27:36.067171 14499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:27:36.067229 14493 server_base.cc:1034] running on GCE node
W20260620 04:27:36.067176 14501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:27:36.067792 14493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:27:36.068467 14493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:27:36.069643 14493 hybrid_clock.cc:648] HybridClock initialized: now 1781929656069621 us; error 35 us; skew 500 ppm
Jun 20 04:27:36 dist-test-slave-nk6d krb5kdc[14118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929656, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:27:36.072711 14493 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 04:27:36.073643 14493 webserver.cc:460] Webserver started at http://127.0.98.3:42903/ using document root <none> and password file <none>
I20260620 04:27:36.074198 14493 fs_manager.cc:359] Metadata directory not provided
I20260620 04:27:36.074262 14493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:27:36.074450 14493 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:27:36.076520 14493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-2/data/instance:
uuid: "a7cb00dde55d424d9dc7ee1fc139c6dc"
format_stamp: "Formatted at 2026-06-20 04:27:36 on dist-test-slave-nk6d"
server_key: "dedf2077f899b792f7a0bb8539d31e1e"
server_key_iv: "f2b819fcd140ae7eed49935e59d790d3"
server_key_version: "encryptionkey@0"
I20260620 04:27:36.077046 14493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a7cb00dde55d424d9dc7ee1fc139c6dc"
format_stamp: "Formatted at 2026-06-20 04:27:36 on dist-test-slave-nk6d"
server_key: "dedf2077f899b792f7a0bb8539d31e1e"
server_key_iv: "f2b819fcd140ae7eed49935e59d790d3"
server_key_version: "encryptionkey@0"
I20260620 04:27:36.080704 14493 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:27:36.083099 14508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:27:36.084309 14493 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:27:36.084412 14493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-2/wal
uuid: "a7cb00dde55d424d9dc7ee1fc139c6dc"
format_stamp: "Formatted at 2026-06-20 04:27:36 on dist-test-slave-nk6d"
server_key: "dedf2077f899b792f7a0bb8539d31e1e"
server_key_iv: "f2b819fcd140ae7eed49935e59d790d3"
server_key_version: "encryptionkey@0"
I20260620 04:27:36.084506 14493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:27:36.099279 14493 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:27:36.102231 14493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:27:36.102407 14493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:27:36.102998 14493 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:27:36.103807 14493 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:27:36.103854 14493 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:27:36.103921 14493 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:27:36.103945 14493 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:27:36.112278 14493 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36275
I20260620 04:27:36.112325 14621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36275 every 8 connection(s)
I20260620 04:27:36.113190 14493 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-2/data/info.pb
I20260620 04:27:36.122244 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 14493
I20260620 04:27:36.122362 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-2/wal/instance
I20260620 04:27:36.122633 392 external_mini_cluster.cc:1428] Setting key f4f50a5dd2b39db8dd8a91af13f93434
Jun 20 04:27:36 dist-test-slave-nk6d krb5kdc[14118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929656, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:27:36.126999 14202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:36.114980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38367 (local address 127.0.98.62:36915)
0620 04:27:36.115109 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 04:27:36.115113 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:27:36.116280 (+ 1167us) server_negotiation.cc:407] Connection header received
0620 04:27:36.117460 (+ 1180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:27:36.117463 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:27:36.117515 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:27:36.117593 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:27:36.118830 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:36.119355 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:36.120575 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:36.120788 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:36.120858 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:27:36.123249 (+ 2391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:27:36.123253 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:27:36.123255 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:27:36.123281 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:27:36.124938 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:27:36.125450 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:27:36.125454 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:27:36.125455 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:27:36.125555 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:27:36.125937 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:27:36.125940 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:27:36.125941 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:27:36.126154 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:27:36.126243 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 04:27:36.126734 (+ 491us) server_negotiation.cc:299] Negotiation successful
0620 04:27:36.126883 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 04:27:36.127813 14624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:36.115346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36915 (local address 127.0.98.3:38367)
0620 04:27:36.116093 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:36.116152 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 04:27:36.117222 (+ 1070us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:27:36.117799 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:27:36.117806 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:27:36.118195 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:27:36.118679 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:27:36.118692 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:27:36.119464 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:27:36.119466 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:27:36.120466 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:27:36.120476 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:27:36.120904 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:27:36.120907 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:27:36.121045 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:27:36.121596 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:27:36.121613 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:27:36.123081 (+ 1468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:27:36.125036 (+ 1955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:27:36.125041 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:27:36.125054 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:27:36.125332 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:27:36.125677 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:27:36.125679 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:27:36.125681 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:27:36.125837 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:27:36.126243 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:27:36.126248 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:27:36.126494 (+ 246us) client_negotiation.cc:769] Sending connection context
0620 04:27:36.126648 (+ 154us) client_negotiation.cc:240] Negotiation successful
0620 04:27:36.126912 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":113,"threads_started":1}
I20260620 04:27:36.128898 14622 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36915
I20260620 04:27:36.129132 14622 heartbeater.cc:464] Registering TS with master...
I20260620 04:27:36.129624 14622 heartbeater.cc:511] Master 127.0.98.62:36915 requested a full tablet report, sending...
I20260620 04:27:36.130647 14163 ts_manager.cc:194] Registered new tserver with Master: a7cb00dde55d424d9dc7ee1fc139c6dc (127.0.98.3:36275)
I20260620 04:27:36.131184 14163 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:38367
I20260620 04:27:36.136412 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 04:27:36.144987 14202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:36.138235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42252 (local address 127.0.98.62:36915)
0620 04:27:36.138361 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 04:27:36.138373 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 04:27:36.138498 (+ 125us) server_negotiation.cc:407] Connection header received
0620 04:27:36.138598 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:27:36.138600 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:27:36.138638 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:27:36.138698 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:27:36.139345 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:36.139851 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:36.140654 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:36.140855 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:36.140925 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:27:36.142140 (+ 1215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:27:36.142143 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:27:36.142145 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:27:36.142168 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:27:36.143464 (+ 1296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:27:36.143828 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:27:36.143831 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:27:36.143832 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:27:36.143909 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:27:36.144193 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:27:36.144195 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:27:36.144210 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:27:36.144461 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:27:36.144564 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 04:27:36.144757 (+ 193us) server_negotiation.cc:299] Negotiation successful
0620 04:27:36.144853 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 04:27:36.147921 14163 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42252:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 04:27:36.151042 14163 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 04:27:36.165704 14638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:36.160731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37848 (local address 127.0.98.3:36275)
0620 04:27:36.161149 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0620 04:27:36.161160 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 04:27:36.161400 (+ 240us) server_negotiation.cc:407] Connection header received
0620 04:27:36.161608 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:27:36.161614 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:27:36.161809 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:27:36.161957 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:27:36.162180 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:36.163159 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:36.164496 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:36.165035 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:36.165133 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:27:36.165298 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0620 04:27:36.165442 (+ 144us) server_negotiation.cc:299] Negotiation successful
0620 04:27:36.165540 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":120,"threads_started":1}
I20260620 04:27:36.165695 14637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:36.160430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36275 (local address 127.0.0.1:37848)
0620 04:27:36.161313 (+ 883us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:36.161327 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:27:36.161460 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:27:36.161961 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:27:36.161964 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:27:36.161983 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:27:36.162069 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:27:36.162075 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:27:36.163270 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:27:36.163273 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:27:36.164386 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:27:36.164395 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:27:36.165148 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:27:36.165151 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:27:36.165257 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:27:36.165294 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 04:27:36.165406 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 04:27:36.165541 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":809,"mutex_wait_us":43,"thread_start_us":91,"threads_started":1}
I20260620 04:27:36.165871 14635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:36.159522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43567 (local address 127.0.0.1:51682)
0620 04:27:36.160374 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:36.160388 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:27:36.160533 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:27:36.161302 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:27:36.161309 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:27:36.161359 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:27:36.161528 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:27:36.161536 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:27:36.162970 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:27:36.162973 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:27:36.164118 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:27:36.164126 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:27:36.165354 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:27:36.165357 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:27:36.165524 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:27:36.165578 (+ 54us) client_negotiation.cc:769] Sending connection context
0620 04:27:36.165643 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 04:27:36.165717 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":791,"mutex_wait_us":185,"spinlock_wait_cycles":35200,"thread_start_us":111,"threads_started":1}
I20260620 04:27:36.166203 14634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:36.159620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54958 (local address 127.0.98.1:36255)
0620 04:27:36.160204 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0620 04:27:36.160216 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 04:27:36.160447 (+ 231us) server_negotiation.cc:407] Connection header received
0620 04:27:36.160719 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:27:36.160725 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:27:36.160955 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:27:36.161089 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:27:36.161617 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:36.162536 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:36.164187 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:36.165058 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:36.165139 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:27:36.165288 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 04:27:36.165709 (+ 421us) server_negotiation.cc:299] Negotiation successful
0620 04:27:36.166070 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":80,"threads_started":1}
I20260620 04:27:36.167033 14636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:36.159765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51682 (local address 127.0.98.2:43567)
0620 04:27:36.160400 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0620 04:27:36.160413 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 04:27:36.160465 (+ 52us) server_negotiation.cc:407] Connection header received
0620 04:27:36.160728 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:27:36.160734 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:27:36.160957 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:27:36.161103 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:27:36.161658 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:36.162841 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:36.164246 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:36.165058 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:36.165155 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:27:36.165288 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 04:27:36.166793 (+ 1505us) server_negotiation.cc:299] Negotiation successful
0620 04:27:36.166896 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":80,"threads_started":1}
I20260620 04:27:36.166551 14633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:36.159512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36255 (local address 127.0.0.1:54958)
0620 04:27:36.160305 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:36.160363 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 04:27:36.160535 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:27:36.161312 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:27:36.161315 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:27:36.161334 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:27:36.161526 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:27:36.161533 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:27:36.162655 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:27:36.162659 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:27:36.164044 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:27:36.164055 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:27:36.165371 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:27:36.165374 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:27:36.165523 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:27:36.165579 (+ 56us) client_negotiation.cc:769] Sending connection context
0620 04:27:36.166359 (+ 780us) client_negotiation.cc:240] Negotiation successful
0620 04:27:36.166435 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":606,"mutex_wait_us":74,"spinlock_wait_cycles":6400,"thread_start_us":102,"threads_started":1}
I20260620 04:27:36.168946 14556 tablet_service.cc:1467] Processing CreateTablet for tablet c89d76c347dd49c1a4defb0c53c9a3fb (DEFAULT_TABLE table=test-table [id=77d4fd48ecec443ab6622073431c36c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:27:36.169507 14422 tablet_service.cc:1467] Processing CreateTablet for tablet c89d76c347dd49c1a4defb0c53c9a3fb (DEFAULT_TABLE table=test-table [id=77d4fd48ecec443ab6622073431c36c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:27:36.169945 14556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c89d76c347dd49c1a4defb0c53c9a3fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:27:36.170387 14422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c89d76c347dd49c1a4defb0c53c9a3fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:27:36.176258 14639 tablet_bootstrap.cc:492] T c89d76c347dd49c1a4defb0c53c9a3fb P a7cb00dde55d424d9dc7ee1fc139c6dc: Bootstrap starting.
I20260620 04:27:36.176258 14640 tablet_bootstrap.cc:492] T c89d76c347dd49c1a4defb0c53c9a3fb P 126db552d7e645d597ecbab7867c8368: Bootstrap starting.
I20260620 04:27:36.178339 14639 tablet_bootstrap.cc:654] T c89d76c347dd49c1a4defb0c53c9a3fb P a7cb00dde55d424d9dc7ee1fc139c6dc: Neither blocks nor log segments found. Creating new log.
I20260620 04:27:36.178280 14288 tablet_service.cc:1467] Processing CreateTablet for tablet c89d76c347dd49c1a4defb0c53c9a3fb (DEFAULT_TABLE table=test-table [id=77d4fd48ecec443ab6622073431c36c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:27:36.179353 14639 log.cc:824] T c89d76c347dd49c1a4defb0c53c9a3fb P a7cb00dde55d424d9dc7ee1fc139c6dc: Log is configured to *not* fsync() on all Append() calls
I20260620 04:27:36.179401 14288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c89d76c347dd49c1a4defb0c53c9a3fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:27:36.179811 14640 tablet_bootstrap.cc:654] T c89d76c347dd49c1a4defb0c53c9a3fb P 126db552d7e645d597ecbab7867c8368: Neither blocks nor log segments found. Creating new log.
I20260620 04:27:36.180655 14640 log.cc:824] T c89d76c347dd49c1a4defb0c53c9a3fb P 126db552d7e645d597ecbab7867c8368: Log is configured to *not* fsync() on all Append() calls
I20260620 04:27:36.181732 14639 tablet_bootstrap.cc:492] T c89d76c347dd49c1a4defb0c53c9a3fb P a7cb00dde55d424d9dc7ee1fc139c6dc: No bootstrap required, opened a new log
I20260620 04:27:36.181921 14639 ts_tablet_manager.cc:1397] T c89d76c347dd49c1a4defb0c53c9a3fb P a7cb00dde55d424d9dc7ee1fc139c6dc: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260620 04:27:36.182937 14640 tablet_bootstrap.cc:492] T c89d76c347dd49c1a4defb0c53c9a3fb P 126db552d7e645d597ecbab7867c8368: No bootstrap required, opened a new log
I20260620 04:27:36.183125 14640 ts_tablet_manager.cc:1397] T c89d76c347dd49c1a4defb0c53c9a3fb P 126db552d7e645d597ecbab7867c8368: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260620 04:27:36.185042 14639 raft_consensus.cc:348] T c89d76c347dd49c1a4defb0c53c9a3fb P a7cb00dde55d424d9dc7ee1fc139c6dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0a07d8cbfd141e49fcce98f279a8ebc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36255 } } peers { permanent_uuid: "126db552d7e645d597ecbab7867c8368" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43567 } } peers { permanent_uuid: "a7cb00dde55d424d9dc7ee1fc139c6dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36275 } }
I20260620 04:27:36.185308 14639 raft_consensus.cc:374] T c89d76c347dd49c1a4defb0c53c9a3fb P a7cb00dde55d424d9dc7ee1fc139c6dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:27:36.185362 14639 raft_consensus.cc:729] T c89d76c347dd49c1a4defb0c53c9a3fb P a7cb00dde55d424d9dc7ee1fc139c6dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7cb00dde55d424d9dc7ee1fc139c6dc, State: Initialized, Role: FOLLOWER
I20260620 04:27:36.185757 14639 consensus_queue.cc:260] T c89d76c347dd49c1a4defb0c53c9a3fb P a7cb00dde55d424d9dc7ee1fc139c6dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0a07d8cbfd141e49fcce98f279a8ebc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36255 } } peers { permanent_uuid: "126db552d7e645d597ecbab7867c8368" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43567 } } peers { permanent_uuid: "a7cb00dde55d424d9dc7ee1fc139c6dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36275 } }
I20260620 04:27:36.186510 14622 heartbeater.cc:503] Master 127.0.98.62:36915 was elected leader, sending a full tablet report...
I20260620 04:27:36.186750 14639 ts_tablet_manager.cc:1428] T c89d76c347dd49c1a4defb0c53c9a3fb P a7cb00dde55d424d9dc7ee1fc139c6dc: Time spent starting tablet: real 0.005s user 0.000s sys 0.005s
I20260620 04:27:36.186965 14640 raft_consensus.cc:348] T c89d76c347dd49c1a4defb0c53c9a3fb P 126db552d7e645d597ecbab7867c8368 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0a07d8cbfd141e49fcce98f279a8ebc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36255 } } peers { permanent_uuid: "126db552d7e645d597ecbab7867c8368" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43567 } } peers { permanent_uuid: "a7cb00dde55d424d9dc7ee1fc139c6dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36275 } }
I20260620 04:27:36.187225 14640 raft_consensus.cc:374] T c89d76c347dd49c1a4defb0c53c9a3fb P 126db552d7e645d597ecbab7867c8368 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:27:36.187275 14640 raft_consensus.cc:729] T c89d76c347dd49c1a4defb0c53c9a3fb P 126db552d7e645d597ecbab7867c8368 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 126db552d7e645d597ecbab7867c8368, State: Initialized, Role: FOLLOWER
I20260620 04:27:36.187347 14644 tablet_bootstrap.cc:492] T c89d76c347dd49c1a4defb0c53c9a3fb P a0a07d8cbfd141e49fcce98f279a8ebc: Bootstrap starting.
I20260620 04:27:36.187682 14640 consensus_queue.cc:260] T c89d76c347dd49c1a4defb0c53c9a3fb P 126db552d7e645d597ecbab7867c8368 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0a07d8cbfd141e49fcce98f279a8ebc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36255 } } peers { permanent_uuid: "126db552d7e645d597ecbab7867c8368" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43567 } } peers { permanent_uuid: "a7cb00dde55d424d9dc7ee1fc139c6dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36275 } }
I20260620 04:27:36.188635 14640 ts_tablet_manager.cc:1428] T c89d76c347dd49c1a4defb0c53c9a3fb P 126db552d7e645d597ecbab7867c8368: Time spent starting tablet: real 0.005s user 0.003s sys 0.002s
I20260620 04:27:36.189042 14488 heartbeater.cc:503] Master 127.0.98.62:36915 was elected leader, sending a full tablet report...
I20260620 04:27:36.190407 14644 tablet_bootstrap.cc:654] T c89d76c347dd49c1a4defb0c53c9a3fb P a0a07d8cbfd141e49fcce98f279a8ebc: Neither blocks nor log segments found. Creating new log.
I20260620 04:27:36.191566 14644 log.cc:824] T c89d76c347dd49c1a4defb0c53c9a3fb P a0a07d8cbfd141e49fcce98f279a8ebc: Log is configured to *not* fsync() on all Append() calls
I20260620 04:27:36.193586 14644 tablet_bootstrap.cc:492] T c89d76c347dd49c1a4defb0c53c9a3fb P a0a07d8cbfd141e49fcce98f279a8ebc: No bootstrap required, opened a new log
I20260620 04:27:36.193755 14644 ts_tablet_manager.cc:1397] T c89d76c347dd49c1a4defb0c53c9a3fb P a0a07d8cbfd141e49fcce98f279a8ebc: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260620 04:27:36.196620 14644 raft_consensus.cc:348] T c89d76c347dd49c1a4defb0c53c9a3fb P a0a07d8cbfd141e49fcce98f279a8ebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0a07d8cbfd141e49fcce98f279a8ebc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36255 } } peers { permanent_uuid: "126db552d7e645d597ecbab7867c8368" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43567 } } peers { permanent_uuid: "a7cb00dde55d424d9dc7ee1fc139c6dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36275 } }
I20260620 04:27:36.196846 14644 raft_consensus.cc:374] T c89d76c347dd49c1a4defb0c53c9a3fb P a0a07d8cbfd141e49fcce98f279a8ebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:27:36.196882 14644 raft_consensus.cc:729] T c89d76c347dd49c1a4defb0c53c9a3fb P a0a07d8cbfd141e49fcce98f279a8ebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0a07d8cbfd141e49fcce98f279a8ebc, State: Initialized, Role: FOLLOWER
I20260620 04:27:36.197227 14644 consensus_queue.cc:260] T c89d76c347dd49c1a4defb0c53c9a3fb P a0a07d8cbfd141e49fcce98f279a8ebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0a07d8cbfd141e49fcce98f279a8ebc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36255 } } peers { permanent_uuid: "126db552d7e645d597ecbab7867c8368" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43567 } } peers { permanent_uuid: "a7cb00dde55d424d9dc7ee1fc139c6dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36275 } }
I20260620 04:27:36.197820 14354 heartbeater.cc:503] Master 127.0.98.62:36915 was elected leader, sending a full tablet report...
I20260620 04:27:36.198132 14644 ts_tablet_manager.cc:1428] T c89d76c347dd49c1a4defb0c53c9a3fb P a0a07d8cbfd141e49fcce98f279a8ebc: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
W20260620 04:27:36.210114 14355 tablet.cc:2369] T c89d76c347dd49c1a4defb0c53c9a3fb P a0a07d8cbfd141e49fcce98f279a8ebc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 04:27:36.364182 14623 tablet.cc:2369] T c89d76c347dd49c1a4defb0c53c9a3fb P a7cb00dde55d424d9dc7ee1fc139c6dc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 04:27:36.409564 14489 tablet.cc:2369] T c89d76c347dd49c1a4defb0c53c9a3fb P 126db552d7e645d597ecbab7867c8368: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 04:27:36.440143 14647 raft_consensus.cc:482] T c89d76c347dd49c1a4defb0c53c9a3fb P a0a07d8cbfd141e49fcce98f279a8ebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:27:36.440346 14647 raft_consensus.cc:504] T c89d76c347dd49c1a4defb0c53c9a3fb P a0a07d8cbfd141e49fcce98f279a8ebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0a07d8cbfd141e49fcce98f279a8ebc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36255 } } peers { permanent_uuid: "126db552d7e645d597ecbab7867c8368" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43567 } } peers { permanent_uuid: "a7cb00dde55d424d9dc7ee1fc139c6dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36275 } }
I20260620 04:27:36.441584 14647 leader_election.cc:288] T c89d76c347dd49c1a4defb0c53c9a3fb P a0a07d8cbfd141e49fcce98f279a8ebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 126db552d7e645d597ecbab7867c8368 (127.0.98.2:43567), a7cb00dde55d424d9dc7ee1fc139c6dc (127.0.98.3:36275)
I20260620 04:27:36.446888 14636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:36.441907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48535 (local address 127.0.98.2:43567)
0620 04:27:36.442047 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 04:27:36.442052 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:27:36.442231 (+ 179us) server_negotiation.cc:407] Connection header received
0620 04:27:36.442388 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:27:36.442391 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:27:36.442442 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:27:36.442536 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:27:36.442913 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:36.443683 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:36.445512 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:36.446133 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:36.446239 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:27:36.446304 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 04:27:36.446661 (+ 357us) server_negotiation.cc:299] Negotiation successful
0620 04:27:36.446732 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 04:27:36.446866 14638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:36.442009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46583 (local address 127.0.98.3:36275)
0620 04:27:36.442159 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 04:27:36.442163 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:27:36.442263 (+ 100us) server_negotiation.cc:407] Connection header received
0620 04:27:36.442388 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:27:36.442391 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:27:36.442442 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:27:36.442534 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:27:36.442903 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:36.443700 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:36.445512 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:36.446133 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:36.446239 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:27:36.446303 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 04:27:36.446657 (+ 354us) server_negotiation.cc:299] Negotiation successful
0620 04:27:36.446732 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 04:27:36.446914 14649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:36.441848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36275 (local address 127.0.98.1:46583)
0620 04:27:36.442185 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:36.442195 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 04:27:36.442291 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:27:36.442552 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:27:36.442555 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:27:36.442581 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:27:36.442776 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:27:36.442781 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:27:36.443803 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:27:36.443807 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:27:36.445375 (+ 1568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:27:36.445386 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:27:36.446278 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:27:36.446282 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:27:36.446484 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:27:36.446559 (+ 75us) client_negotiation.cc:769] Sending connection context
0620 04:27:36.446635 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 04:27:36.446741 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"mutex_wait_us":262,"spinlock_wait_cycles":14336,"thread_start_us":111,"threads_started":1}
I20260620 04:27:36.446918 14648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:36.441781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43567 (local address 127.0.98.1:48535)
0620 04:27:36.442140 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:36.442155 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 04:27:36.442291 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:27:36.442552 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:27:36.442555 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:27:36.442578 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:27:36.442769 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:27:36.442776 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:27:36.443803 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:27:36.443807 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:27:36.445375 (+ 1568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:27:36.445386 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:27:36.446278 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:27:36.446282 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:27:36.446513 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:27:36.446552 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 04:27:36.446635 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 04:27:36.446757 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"mutex_wait_us":230,"spinlock_wait_cycles":15232,"thread_start_us":101,"threads_started":1}
I20260620 04:27:36.447614 14442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c89d76c347dd49c1a4defb0c53c9a3fb" candidate_uuid: "a0a07d8cbfd141e49fcce98f279a8ebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "126db552d7e645d597ecbab7867c8368" is_pre_election: true
I20260620 04:27:36.447613 14576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c89d76c347dd49c1a4defb0c53c9a3fb" candidate_uuid: "a0a07d8cbfd141e49fcce98f279a8ebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7cb00dde55d424d9dc7ee1fc139c6dc" is_pre_election: true
I20260620 04:27:36.447889 14442 raft_consensus.cc:2454] T c89d76c347dd49c1a4defb0c53c9a3fb P 126db552d7e645d597ecbab7867c8368 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0a07d8cbfd141e49fcce98f279a8ebc in term 0.
I20260620 04:27:36.447896 14576 raft_consensus.cc:2454] T c89d76c347dd49c1a4defb0c53c9a3fb P a7cb00dde55d424d9dc7ee1fc139c6dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0a07d8cbfd141e49fcce98f279a8ebc in term 0.
I20260620 04:27:36.448385 14242 leader_election.cc:302] T c89d76c347dd49c1a4defb0c53c9a3fb P a0a07d8cbfd141e49fcce98f279a8ebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0a07d8cbfd141e49fcce98f279a8ebc, a7cb00dde55d424d9dc7ee1fc139c6dc; no voters:
I20260620 04:27:36.448657 14647 raft_consensus.cc:2781] T c89d76c347dd49c1a4defb0c53c9a3fb P a0a07d8cbfd141e49fcce98f279a8ebc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 04:27:36.448738 14647 raft_consensus.cc:482] T c89d76c347dd49c1a4defb0c53c9a3fb P a0a07d8cbfd141e49fcce98f279a8ebc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 04:27:36.448778 14647 raft_consensus.cc:3037] T c89d76c347dd49c1a4defb0c53c9a3fb P a0a07d8cbfd141e49fcce98f279a8ebc [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:27:36.449956 14647 raft_consensus.cc:504] T c89d76c347dd49c1a4defb0c53c9a3fb P a0a07d8cbfd141e49fcce98f279a8ebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0a07d8cbfd141e49fcce98f279a8ebc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36255 } } peers { permanent_uuid: "126db552d7e645d597ecbab7867c8368" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43567 } } peers { permanent_uuid: "a7cb00dde55d424d9dc7ee1fc139c6dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36275 } }
I20260620 04:27:36.450294 14647 leader_election.cc:288] T c89d76c347dd49c1a4defb0c53c9a3fb P a0a07d8cbfd141e49fcce98f279a8ebc [CANDIDATE]: Term 1 election: Requested vote from peers 126db552d7e645d597ecbab7867c8368 (127.0.98.2:43567), a7cb00dde55d424d9dc7ee1fc139c6dc (127.0.98.3:36275)
I20260620 04:27:36.450591 14442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c89d76c347dd49c1a4defb0c53c9a3fb" candidate_uuid: "a0a07d8cbfd141e49fcce98f279a8ebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "126db552d7e645d597ecbab7867c8368"
I20260620 04:27:36.450700 14576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c89d76c347dd49c1a4defb0c53c9a3fb" candidate_uuid: "a0a07d8cbfd141e49fcce98f279a8ebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7cb00dde55d424d9dc7ee1fc139c6dc"
I20260620 04:27:36.450706 14442 raft_consensus.cc:3037] T c89d76c347dd49c1a4defb0c53c9a3fb P 126db552d7e645d597ecbab7867c8368 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:27:36.450821 14576 raft_consensus.cc:3037] T c89d76c347dd49c1a4defb0c53c9a3fb P a7cb00dde55d424d9dc7ee1fc139c6dc [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:27:36.451902 14576 raft_consensus.cc:2454] T c89d76c347dd49c1a4defb0c53c9a3fb P a7cb00dde55d424d9dc7ee1fc139c6dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0a07d8cbfd141e49fcce98f279a8ebc in term 1.
I20260620 04:27:36.451900 14442 raft_consensus.cc:2454] T c89d76c347dd49c1a4defb0c53c9a3fb P 126db552d7e645d597ecbab7867c8368 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0a07d8cbfd141e49fcce98f279a8ebc in term 1.
I20260620 04:27:36.452165 14242 leader_election.cc:302] T c89d76c347dd49c1a4defb0c53c9a3fb P a0a07d8cbfd141e49fcce98f279a8ebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0a07d8cbfd141e49fcce98f279a8ebc, a7cb00dde55d424d9dc7ee1fc139c6dc; no voters:
I20260620 04:27:36.452323 14647 raft_consensus.cc:2781] T c89d76c347dd49c1a4defb0c53c9a3fb P a0a07d8cbfd141e49fcce98f279a8ebc [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:27:36.452528 14647 raft_consensus.cc:686] T c89d76c347dd49c1a4defb0c53c9a3fb P a0a07d8cbfd141e49fcce98f279a8ebc [term 1 LEADER]: Becoming Leader. State: Replica: a0a07d8cbfd141e49fcce98f279a8ebc, State: Running, Role: LEADER
I20260620 04:27:36.452833 14647 consensus_queue.cc:237] T c89d76c347dd49c1a4defb0c53c9a3fb P a0a07d8cbfd141e49fcce98f279a8ebc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0a07d8cbfd141e49fcce98f279a8ebc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36255 } } peers { permanent_uuid: "126db552d7e645d597ecbab7867c8368" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43567 } } peers { permanent_uuid: "a7cb00dde55d424d9dc7ee1fc139c6dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36275 } }
I20260620 04:27:36.455921 14161 catalog_manager.cc:5495] T c89d76c347dd49c1a4defb0c53c9a3fb P a0a07d8cbfd141e49fcce98f279a8ebc reported cstate change: term changed from 0 to 1, leader changed from <none> to a0a07d8cbfd141e49fcce98f279a8ebc (127.0.98.1). New cstate: current_term: 1 leader_uuid: "a0a07d8cbfd141e49fcce98f279a8ebc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0a07d8cbfd141e49fcce98f279a8ebc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36255 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "126db552d7e645d597ecbab7867c8368" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43567 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a7cb00dde55d424d9dc7ee1fc139c6dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36275 } health_report { overall_health: UNKNOWN } } }
I20260620 04:27:36.529155 14634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:36.525651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54962 (local address 127.0.98.1:36255)
0620 04:27:36.525768 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 04:27:36.525771 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:27:36.525799 (+ 28us) server_negotiation.cc:407] Connection header received
0620 04:27:36.525882 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:27:36.525884 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:27:36.525940 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:27:36.526024 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:27:36.526212 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:36.526704 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:36.527507 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:36.527760 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:36.527849 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:27:36.528178 (+ 329us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:27:36.528674 (+ 496us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:27:36.528789 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 04:27:36.528971 (+ 182us) server_negotiation.cc:299] Negotiation successful
0620 04:27:36.529035 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 04:27:36.535113 14442 raft_consensus.cc:1261] T c89d76c347dd49c1a4defb0c53c9a3fb P 126db552d7e645d597ecbab7867c8368 [term 1 FOLLOWER]: Refusing update from remote peer a0a07d8cbfd141e49fcce98f279a8ebc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:27:36.535113 14576 raft_consensus.cc:1261] T c89d76c347dd49c1a4defb0c53c9a3fb P a7cb00dde55d424d9dc7ee1fc139c6dc [term 1 FOLLOWER]: Refusing update from remote peer a0a07d8cbfd141e49fcce98f279a8ebc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:27:36.535784 14647 consensus_queue.cc:1035] T c89d76c347dd49c1a4defb0c53c9a3fb P a0a07d8cbfd141e49fcce98f279a8ebc [LEADER]: Connected to new peer: Peer: permanent_uuid: "126db552d7e645d597ecbab7867c8368" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:27:36.535980 14650 consensus_queue.cc:1035] T c89d76c347dd49c1a4defb0c53c9a3fb P a0a07d8cbfd141e49fcce98f279a8ebc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7cb00dde55d424d9dc7ee1fc139c6dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36275 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:27:36.546449 14655 mvcc.cc:205] Tried to move back new op lower bound from 7298783873160581120 to 7298783872833175552. Current Snapshot: MvccSnapshot[applied={T|T < 7298783873160581120 or (T in {7298783873160581120})}]
I20260620 04:27:36.553754 14161 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42252:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:27:36.553932 14161 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42252:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:27:36.556129 14161 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a87a20f134ed4ae9a447a40942f3d7ea: Sending DeleteTablet for 3 replicas of tablet c89d76c347dd49c1a4defb0c53c9a3fb
I20260620 04:27:36.556880 14556 tablet_service.cc:1514] Processing DeleteTablet for tablet c89d76c347dd49c1a4defb0c53c9a3fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:27:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37848
I20260620 04:27:36.556893 14422 tablet_service.cc:1514] Processing DeleteTablet for tablet c89d76c347dd49c1a4defb0c53c9a3fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:27:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51682
I20260620 04:27:36.557013 14288 tablet_service.cc:1514] Processing DeleteTablet for tablet c89d76c347dd49c1a4defb0c53c9a3fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:27:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54958
I20260620 04:27:36.557392 14662 tablet_replica.cc:330] T c89d76c347dd49c1a4defb0c53c9a3fb P 126db552d7e645d597ecbab7867c8368: stopping tablet replica
I20260620 04:27:36.557600 14664 tablet_replica.cc:330] T c89d76c347dd49c1a4defb0c53c9a3fb P a0a07d8cbfd141e49fcce98f279a8ebc: stopping tablet replica
I20260620 04:27:36.557634 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 14225
I20260620 04:27:36.557885 14664 raft_consensus.cc:2229] T c89d76c347dd49c1a4defb0c53c9a3fb P a0a07d8cbfd141e49fcce98f279a8ebc [term 1 LEADER]: Raft consensus shutting down.
I20260620 04:27:36.558303 14663 tablet_replica.cc:330] T c89d76c347dd49c1a4defb0c53c9a3fb P a7cb00dde55d424d9dc7ee1fc139c6dc: stopping tablet replica
I20260620 04:27:36.558568 14663 raft_consensus.cc:2229] T c89d76c347dd49c1a4defb0c53c9a3fb P a7cb00dde55d424d9dc7ee1fc139c6dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:27:36.558841 14663 raft_consensus.cc:2258] T c89d76c347dd49c1a4defb0c53c9a3fb P a7cb00dde55d424d9dc7ee1fc139c6dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:27:36.560222 14663 ts_tablet_manager.cc:1905] T c89d76c347dd49c1a4defb0c53c9a3fb P a7cb00dde55d424d9dc7ee1fc139c6dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:27:36.562318 14663 ts_tablet_manager.cc:1918] T c89d76c347dd49c1a4defb0c53c9a3fb P a7cb00dde55d424d9dc7ee1fc139c6dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:27:36.562397 14663 log.cc:1196] T c89d76c347dd49c1a4defb0c53c9a3fb P a7cb00dde55d424d9dc7ee1fc139c6dc: Deleting WAL directory at /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-2/wal/wals/c89d76c347dd49c1a4defb0c53c9a3fb
I20260620 04:27:36.562709 14663 ts_tablet_manager.cc:1939] T c89d76c347dd49c1a4defb0c53c9a3fb P a7cb00dde55d424d9dc7ee1fc139c6dc: Deleting consensus metadata
I20260620 04:27:36.562716 14662 raft_consensus.cc:2229] T c89d76c347dd49c1a4defb0c53c9a3fb P 126db552d7e645d597ecbab7867c8368 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:27:36.563138 14662 raft_consensus.cc:2258] T c89d76c347dd49c1a4defb0c53c9a3fb P 126db552d7e645d597ecbab7867c8368 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:27:36.564196 14148 catalog_manager.cc:4841] TS a7cb00dde55d424d9dc7ee1fc139c6dc (127.0.98.3:36275): tablet c89d76c347dd49c1a4defb0c53c9a3fb (table test-table [id=77d4fd48ecec443ab6622073431c36c4]) successfully deleted
W20260620 04:27:36.564476 14148 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36255 (error 108)
I20260620 04:27:36.564605 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 14359
I20260620 04:27:36.565930 14633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:36.565633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36255 (local address 127.0.0.1:54972)
0620 04:27:36.565768 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:36.565843 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 04:27:36.566080 14148 catalog_manager.cc:4584] TS a0a07d8cbfd141e49fcce98f279a8ebc (127.0.98.1:36255): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c89d76c347dd49c1a4defb0c53c9a3fb: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36255: connect: Connection refused (error 111)
I20260620 04:27:36.570547 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 14493
I20260620 04:27:36.571493 14633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:36.570712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43567 (local address 127.0.0.1:51684)
0620 04:27:36.571325 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:36.571392 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:43567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":519}
I20260620 04:27:36.575286 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 14131
I20260620 04:27:36.580612 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36915
--webserver_interface=127.0.98.62
--webserver_port=35683
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36915
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:27:36.688023 14666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:27:36.688292 14666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:27:36.688351 14666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:27:36.692369 14666 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:27:36.692430 14666 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:27:36.692456 14666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:27:36.692492 14666 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:27:36.692521 14666 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:27:36.697731 14666 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36915
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36915
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=35683
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14666
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:27:36.698741 14666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:27:36.699731 14666 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:27:36.711856 14672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:27:36.711918 14675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:27:36.712078 14666 server_base.cc:1034] running on GCE node
W20260620 04:27:36.711856 14673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:27:36.712809 14666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:27:36.713934 14666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:27:36.715111 14666 hybrid_clock.cc:648] HybridClock initialized: now 1781929656715091 us; error 35 us; skew 500 ppm
Jun 20 04:27:36 dist-test-slave-nk6d krb5kdc[14118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929656, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:27:36.717933 14666 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 04:27:36.718792 14666 webserver.cc:460] Webserver started at http://127.0.98.62:35683/ using document root <none> and password file <none>
I20260620 04:27:36.719326 14666 fs_manager.cc:359] Metadata directory not provided
I20260620 04:27:36.719424 14666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:27:36.723265 14666 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260620 04:27:36.725425 14682 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:27:36.726590 14666 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 04:27:36.726699 14666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/master-0/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/master-0/wal
uuid: "a87a20f134ed4ae9a447a40942f3d7ea"
format_stamp: "Formatted at 2026-06-20 04:27:35 on dist-test-slave-nk6d"
server_key: "812718ef149fdf619ec957fc50632200"
server_key_iv: "1502d5d4f62daadbaabd2bdeee1d2496"
server_key_version: "encryptionkey@0"
I20260620 04:27:36.727262 14666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:27:36.740357 14666 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:27:36.743418 14666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:27:36.743618 14666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:27:36.750749 14666 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36915
I20260620 04:27:36.751066 14734 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36915 every 8 connection(s)
I20260620 04:27:36.751832 14666 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:27:36.756096 14735 sys_catalog.cc:263] Verifying existing consensus state
I20260620 04:27:36.756276 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 14666
I20260620 04:27:36.756666 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:36255
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:36915
--webserver_port=36093
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 04:27:36.757876 14735 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a87a20f134ed4ae9a447a40942f3d7ea: Bootstrap starting.
I20260620 04:27:36.780541 14735 log.cc:824] T 00000000000000000000000000000000 P a87a20f134ed4ae9a447a40942f3d7ea: Log is configured to *not* fsync() on all Append() calls
I20260620 04:27:36.791736 14735 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a87a20f134ed4ae9a447a40942f3d7ea: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 04:27:36.792202 14735 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a87a20f134ed4ae9a447a40942f3d7ea: Bootstrap complete.
I20260620 04:27:36.797451 14735 raft_consensus.cc:348] T 00000000000000000000000000000000 P a87a20f134ed4ae9a447a40942f3d7ea [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a87a20f134ed4ae9a447a40942f3d7ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36915 } }
I20260620 04:27:36.797953 14735 raft_consensus.cc:729] T 00000000000000000000000000000000 P a87a20f134ed4ae9a447a40942f3d7ea [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a87a20f134ed4ae9a447a40942f3d7ea, State: Initialized, Role: FOLLOWER
I20260620 04:27:36.798542 14735 consensus_queue.cc:260] T 00000000000000000000000000000000 P a87a20f134ed4ae9a447a40942f3d7ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a87a20f134ed4ae9a447a40942f3d7ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36915 } }
I20260620 04:27:36.798691 14735 raft_consensus.cc:388] T 00000000000000000000000000000000 P a87a20f134ed4ae9a447a40942f3d7ea [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:27:36.798763 14735 raft_consensus.cc:482] T 00000000000000000000000000000000 P a87a20f134ed4ae9a447a40942f3d7ea [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:27:36.798867 14735 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a87a20f134ed4ae9a447a40942f3d7ea [term 1 FOLLOWER]: Advancing to term 2
I20260620 04:27:36.801268 14735 raft_consensus.cc:504] T 00000000000000000000000000000000 P a87a20f134ed4ae9a447a40942f3d7ea [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a87a20f134ed4ae9a447a40942f3d7ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36915 } }
I20260620 04:27:36.801626 14735 leader_election.cc:302] T 00000000000000000000000000000000 P a87a20f134ed4ae9a447a40942f3d7ea [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a87a20f134ed4ae9a447a40942f3d7ea; no voters:
I20260620 04:27:36.801975 14735 leader_election.cc:288] T 00000000000000000000000000000000 P a87a20f134ed4ae9a447a40942f3d7ea [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 04:27:36.802111 14739 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a87a20f134ed4ae9a447a40942f3d7ea [term 2 FOLLOWER]: Leader election won for term 2
I20260620 04:27:36.802366 14739 raft_consensus.cc:686] T 00000000000000000000000000000000 P a87a20f134ed4ae9a447a40942f3d7ea [term 2 LEADER]: Becoming Leader. State: Replica: a87a20f134ed4ae9a447a40942f3d7ea, State: Running, Role: LEADER
I20260620 04:27:36.802685 14739 consensus_queue.cc:237] T 00000000000000000000000000000000 P a87a20f134ed4ae9a447a40942f3d7ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a87a20f134ed4ae9a447a40942f3d7ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36915 } }
I20260620 04:27:36.803226 14735 sys_catalog.cc:564] T 00000000000000000000000000000000 P a87a20f134ed4ae9a447a40942f3d7ea [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:27:36.804092 14740 sys_catalog.cc:455] T 00000000000000000000000000000000 P a87a20f134ed4ae9a447a40942f3d7ea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a87a20f134ed4ae9a447a40942f3d7ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a87a20f134ed4ae9a447a40942f3d7ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36915 } } }
I20260620 04:27:36.804235 14740 sys_catalog.cc:458] T 00000000000000000000000000000000 P a87a20f134ed4ae9a447a40942f3d7ea [sys.catalog]: This master's current role is: LEADER
I20260620 04:27:36.804092 14741 sys_catalog.cc:455] T 00000000000000000000000000000000 P a87a20f134ed4ae9a447a40942f3d7ea [sys.catalog]: SysCatalogTable state changed. Reason: New leader a87a20f134ed4ae9a447a40942f3d7ea. Latest consensus state: current_term: 2 leader_uuid: "a87a20f134ed4ae9a447a40942f3d7ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a87a20f134ed4ae9a447a40942f3d7ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36915 } } }
I20260620 04:27:36.804641 14741 sys_catalog.cc:458] T 00000000000000000000000000000000 P a87a20f134ed4ae9a447a40942f3d7ea [sys.catalog]: This master's current role is: LEADER
I20260620 04:27:36.804960 14748 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:27:36.809595 14748 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:27:36.810206 14748 catalog_manager.cc:1259] Loaded cluster ID: 676c7b67d90c4e8a855460a0f74a3aa4
I20260620 04:27:36.810264 14748 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:27:36.811658 14748 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260620 04:27:36.816411 14748 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260620 04:27:36.821921 14748 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:27:36.822685 14748 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a87a20f134ed4ae9a447a40942f3d7ea: Loaded TSK: 0
I20260620 04:27:36.823899 14748 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 04:27:36.880498 14737 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:27:36.880766 14737 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:27:36.880811 14737 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:27:36.884824 14737 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:27:36.884874 14737 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:27:36.884936 14737 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 04:27:36.890077 14737 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:36255
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=36093
--tserver_master_addrs=127.0.98.62:36915
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14737
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:27:36.891038 14737 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:27:36.892030 14737 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:27:36.901049 14762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:27:36.901051 14763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:27:36.901386 14737 server_base.cc:1034] running on GCE node
W20260620 04:27:36.901502 14765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:27:36.901883 14737 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:27:36.902503 14737 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:27:36.903692 14737 hybrid_clock.cc:648] HybridClock initialized: now 1781929656903664 us; error 41 us; skew 500 ppm
Jun 20 04:27:36 dist-test-slave-nk6d krb5kdc[14118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929656, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:27:36.906525 14737 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 04:27:36.907433 14737 webserver.cc:460] Webserver started at http://127.0.98.1:36093/ using document root <none> and password file <none>
I20260620 04:27:36.907990 14737 fs_manager.cc:359] Metadata directory not provided
I20260620 04:27:36.908084 14737 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:27:36.911914 14737 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:27:36.914535 14772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:27:36.915906 14737 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 04:27:36.916075 14737 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-0/wal
uuid: "a0a07d8cbfd141e49fcce98f279a8ebc"
format_stamp: "Formatted at 2026-06-20 04:27:35 on dist-test-slave-nk6d"
server_key: "a118e53c149d76aa58ea1913d93a045b"
server_key_iv: "536eba379b58ca4676c768769cb42be3"
server_key_version: "encryptionkey@0"
I20260620 04:27:36.916638 14737 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:27:36.937700 14737 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:27:36.940771 14737 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:27:36.940944 14737 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:27:36.941565 14737 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:27:36.942659 14779 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 04:27:36.946722 14737 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 04:27:36.946780 14737 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260620 04:27:36.946843 14737 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 04:27:36.948519 14737 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 04:27:36.948562 14737 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260620 04:27:36.948874 14779 tablet_bootstrap.cc:492] T c89d76c347dd49c1a4defb0c53c9a3fb P a0a07d8cbfd141e49fcce98f279a8ebc: Bootstrap starting.
I20260620 04:27:36.958230 14737 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36255
I20260620 04:27:36.958305 14886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36255 every 8 connection(s)
I20260620 04:27:36.959192 14737 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-0/data/info.pb
I20260620 04:27:36.963614 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 14737
I20260620 04:27:36.964087 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:43567
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:36915
--webserver_port=43001
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 04:27:36.966941 14779 log.cc:824] T c89d76c347dd49c1a4defb0c53c9a3fb P a0a07d8cbfd141e49fcce98f279a8ebc: Log is configured to *not* fsync() on all Append() calls
Jun 20 04:27:36 dist-test-slave-nk6d krb5kdc[14118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929656, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:27:36.975363 14889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:36.961223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40293 (local address 127.0.98.62:36915)
0620 04:27:36.961592 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0620 04:27:36.961611 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 04:27:36.961924 (+ 313us) server_negotiation.cc:407] Connection header received
0620 04:27:36.963059 (+ 1135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:27:36.963076 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:27:36.963621 (+ 545us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:27:36.964035 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:27:36.965625 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:36.966623 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:36.967782 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:36.968308 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:36.968486 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:27:36.970751 (+ 2265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:27:36.970757 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:27:36.970768 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:27:36.970797 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:27:36.972358 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:27:36.972827 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:27:36.972832 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:27:36.972836 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:27:36.972949 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:27:36.973250 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:27:36.973253 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:27:36.973255 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:27:36.973606 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:27:36.973730 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 04:27:36.974273 (+ 543us) server_negotiation.cc:299] Negotiation successful
0620 04:27:36.974539 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":107,"threads_started":1}
I20260620 04:27:36.975378 14890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:36.961100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36915 (local address 127.0.98.1:40293)
0620 04:27:36.961750 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:36.961801 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 04:27:36.962605 (+ 804us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:27:36.964105 (+ 1500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:27:36.964114 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:27:36.964797 (+ 683us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:27:36.965453 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:27:36.965475 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:27:36.966772 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:27:36.966775 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:27:36.967665 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:27:36.967675 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:27:36.968461 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:27:36.968465 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:27:36.968653 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:27:36.969198 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:27:36.969214 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:27:36.970563 (+ 1349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:27:36.972468 (+ 1905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:27:36.972474 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:27:36.972487 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:27:36.972723 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:27:36.973036 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:27:36.973039 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:27:36.973041 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:27:36.973165 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:27:36.973723 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:27:36.973729 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:27:36.973985 (+ 256us) client_negotiation.cc:769] Sending connection context
0620 04:27:36.974147 (+ 162us) client_negotiation.cc:240] Negotiation successful
0620 04:27:36.974472 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":144,"threads_started":1}
I20260620 04:27:36.976121 14779 tablet_bootstrap.cc:492] T c89d76c347dd49c1a4defb0c53c9a3fb P a0a07d8cbfd141e49fcce98f279a8ebc: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 04:27:36.976538 14779 tablet_bootstrap.cc:492] T c89d76c347dd49c1a4defb0c53c9a3fb P a0a07d8cbfd141e49fcce98f279a8ebc: Bootstrap complete.
I20260620 04:27:36.977723 14887 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36915
I20260620 04:27:36.977834 14779 ts_tablet_manager.cc:1397] T c89d76c347dd49c1a4defb0c53c9a3fb P a0a07d8cbfd141e49fcce98f279a8ebc: Time spent bootstrapping tablet: real 0.029s user 0.027s sys 0.000s
I20260620 04:27:36.977948 14887 heartbeater.cc:464] Registering TS with master...
I20260620 04:27:36.978416 14887 heartbeater.cc:511] Master 127.0.98.62:36915 requested a full tablet report, sending...
I20260620 04:27:36.980671 14699 ts_manager.cc:194] Registered new tserver with Master: a0a07d8cbfd141e49fcce98f279a8ebc (127.0.98.1:36255)
I20260620 04:27:36.980756 14779 raft_consensus.cc:348] T c89d76c347dd49c1a4defb0c53c9a3fb P a0a07d8cbfd141e49fcce98f279a8ebc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0a07d8cbfd141e49fcce98f279a8ebc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36255 } } peers { permanent_uuid: "126db552d7e645d597ecbab7867c8368" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43567 } } peers { permanent_uuid: "a7cb00dde55d424d9dc7ee1fc139c6dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36275 } }
I20260620 04:27:36.981264 14779 raft_consensus.cc:729] T c89d76c347dd49c1a4defb0c53c9a3fb P a0a07d8cbfd141e49fcce98f279a8ebc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0a07d8cbfd141e49fcce98f279a8ebc, State: Initialized, Role: FOLLOWER
I20260620 04:27:36.981737 14779 consensus_queue.cc:260] T c89d76c347dd49c1a4defb0c53c9a3fb P a0a07d8cbfd141e49fcce98f279a8ebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0a07d8cbfd141e49fcce98f279a8ebc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36255 } } peers { permanent_uuid: "126db552d7e645d597ecbab7867c8368" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43567 } } peers { permanent_uuid: "a7cb00dde55d424d9dc7ee1fc139c6dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36275 } }
I20260620 04:27:36.982875 14779 ts_tablet_manager.cc:1428] T c89d76c347dd49c1a4defb0c53c9a3fb P a0a07d8cbfd141e49fcce98f279a8ebc: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 04:27:36.984419 14699 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:40293
I20260620 04:27:36.987027 14887 heartbeater.cc:503] Master 127.0.98.62:36915 was elected leader, sending a full tablet report...
Jun 20 04:27:36 dist-test-slave-nk6d krb5kdc[14118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929656, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 04:27:36.993541 14895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:36.983665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54974 (local address 127.0.98.1:36255)
0620 04:27:36.984384 (+ 719us) server_negotiation.cc:206] Beginning negotiation
0620 04:27:36.984395 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 04:27:36.984417 (+ 22us) server_negotiation.cc:407] Connection header received
0620 04:27:36.984482 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:27:36.984487 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:27:36.984670 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:27:36.984800 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:27:36.985511 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:36.986210 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:36.987045 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:36.987452 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:36.987551 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:27:36.990084 (+ 2533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:27:36.990093 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:27:36.990103 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:27:36.990124 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:27:36.991508 (+ 1384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:27:36.992139 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:27:36.992144 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:27:36.992145 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:27:36.992229 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:27:36.992545 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:27:36.992548 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:27:36.992549 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:27:36.992802 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:27:36.992929 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 04:27:36.993261 (+ 332us) server_negotiation.cc:299] Negotiation successful
0620 04:27:36.993410 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617,"thread_start_us":101,"threads_started":1}
I20260620 04:27:36.993548 14894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:36.983604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36255 (local address 127.0.0.1:54974)
0620 04:27:36.984033 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:36.984079 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 04:27:36.984200 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:27:36.984821 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:27:36.984827 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:27:36.985264 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:27:36.985396 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:27:36.985405 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:27:36.986314 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:27:36.986318 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:27:36.986929 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:27:36.986938 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:27:36.987540 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:27:36.987557 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:27:36.987663 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:27:36.988261 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:27:36.988276 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:27:36.989901 (+ 1625us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:27:36.991614 (+ 1713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:27:36.991623 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:27:36.991629 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:27:36.992035 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:27:36.992316 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:27:36.992319 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:27:36.992321 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:27:36.992421 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:27:36.992911 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:27:36.992917 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:27:36.993046 (+ 129us) client_negotiation.cc:769] Sending connection context
0620 04:27:36.993178 (+ 132us) client_negotiation.cc:240] Negotiation successful
0620 04:27:36.993383 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":112,"threads_started":1}
I20260620 04:27:36.994629 14821 tablet_service.cc:1514] Processing DeleteTablet for tablet c89d76c347dd49c1a4defb0c53c9a3fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:27:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54974
I20260620 04:27:36.995628 14896 tablet_replica.cc:330] stopping tablet replica
I20260620 04:27:36.995903 14896 raft_consensus.cc:2229] T c89d76c347dd49c1a4defb0c53c9a3fb P a0a07d8cbfd141e49fcce98f279a8ebc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:27:36.996160 14896 raft_consensus.cc:2258] T c89d76c347dd49c1a4defb0c53c9a3fb P a0a07d8cbfd141e49fcce98f279a8ebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:27:36.997087 14896 ts_tablet_manager.cc:1905] T c89d76c347dd49c1a4defb0c53c9a3fb P a0a07d8cbfd141e49fcce98f279a8ebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:27:37.000644 14896 ts_tablet_manager.cc:1918] T c89d76c347dd49c1a4defb0c53c9a3fb P a0a07d8cbfd141e49fcce98f279a8ebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:27:37.000731 14896 log.cc:1196] T c89d76c347dd49c1a4defb0c53c9a3fb P a0a07d8cbfd141e49fcce98f279a8ebc: Deleting WAL directory at /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-0/wal/wals/c89d76c347dd49c1a4defb0c53c9a3fb
I20260620 04:27:37.000964 14896 ts_tablet_manager.cc:1939] T c89d76c347dd49c1a4defb0c53c9a3fb P a0a07d8cbfd141e49fcce98f279a8ebc: Deleting consensus metadata
I20260620 04:27:37.002764 14684 catalog_manager.cc:4841] TS a0a07d8cbfd141e49fcce98f279a8ebc (127.0.98.1:36255): tablet c89d76c347dd49c1a4defb0c53c9a3fb (table test-table [id=77d4fd48ecec443ab6622073431c36c4]) successfully deleted
W20260620 04:27:37.082414 14891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:27:37.082698 14891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:27:37.082739 14891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:27:37.086666 14891 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:27:37.086723 14891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:27:37.086779 14891 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 04:27:37.091846 14891 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:43567
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=43001
--tserver_master_addrs=127.0.98.62:36915
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14891
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:27:37.092832 14891 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:27:37.093788 14891 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:27:37.103065 14901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:27:37.103086 14902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:27:37.103055 14904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:27:37.103407 14891 server_base.cc:1034] running on GCE node
I20260620 04:27:37.103783 14891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:27:37.104467 14891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:27:37.105618 14891 hybrid_clock.cc:648] HybridClock initialized: now 1781929657105598 us; error 35 us; skew 500 ppm
Jun 20 04:27:37 dist-test-slave-nk6d krb5kdc[14118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929657, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:27:37.108332 14891 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 04:27:37.109247 14891 webserver.cc:460] Webserver started at http://127.0.98.2:43001/ using document root <none> and password file <none>
I20260620 04:27:37.109771 14891 fs_manager.cc:359] Metadata directory not provided
I20260620 04:27:37.109844 14891 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:27:37.113687 14891 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260620 04:27:37.115800 14911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:27:37.117101 14891 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:27:37.117211 14891 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-1/wal
uuid: "126db552d7e645d597ecbab7867c8368"
format_stamp: "Formatted at 2026-06-20 04:27:35 on dist-test-slave-nk6d"
server_key: "41c6a4829c42a7f128ddc2edb2c9fa7f"
server_key_iv: "ad041497cafe3adab320c0e4650a0859"
server_key_version: "encryptionkey@0"
I20260620 04:27:37.117745 14891 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:27:37.133519 14891 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:27:37.136637 14891 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:27:37.136806 14891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:27:37.137455 14891 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:27:37.138659 14918 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 04:27:37.142823 14891 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 04:27:37.142884 14891 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.001s
I20260620 04:27:37.142951 14891 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 04:27:37.144623 14891 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 04:27:37.144667 14891 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260620 04:27:37.144946 14918 tablet_bootstrap.cc:492] T c89d76c347dd49c1a4defb0c53c9a3fb P 126db552d7e645d597ecbab7867c8368: Bootstrap starting.
I20260620 04:27:37.150434 14918 log.cc:824] T c89d76c347dd49c1a4defb0c53c9a3fb P 126db552d7e645d597ecbab7867c8368: Log is configured to *not* fsync() on all Append() calls
I20260620 04:27:37.154055 14891 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43567
I20260620 04:27:37.154088 15026 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43567 every 8 connection(s)
I20260620 04:27:37.154863 14891 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-1/data/info.pb
I20260620 04:27:37.158550 14918 tablet_bootstrap.cc:492] T c89d76c347dd49c1a4defb0c53c9a3fb P 126db552d7e645d597ecbab7867c8368: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 04:27:37.158970 14918 tablet_bootstrap.cc:492] T c89d76c347dd49c1a4defb0c53c9a3fb P 126db552d7e645d597ecbab7867c8368: Bootstrap complete.
I20260620 04:27:37.159320 14918 ts_tablet_manager.cc:1397] T c89d76c347dd49c1a4defb0c53c9a3fb P 126db552d7e645d597ecbab7867c8368: Time spent bootstrapping tablet: real 0.015s user 0.011s sys 0.000s
I20260620 04:27:37.161703 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 14891
I20260620 04:27:37.162086 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:36275
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:36915
--webserver_port=42903
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 04:27:37.162163 14918 raft_consensus.cc:348] T c89d76c347dd49c1a4defb0c53c9a3fb P 126db552d7e645d597ecbab7867c8368 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0a07d8cbfd141e49fcce98f279a8ebc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36255 } } peers { permanent_uuid: "126db552d7e645d597ecbab7867c8368" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43567 } } peers { permanent_uuid: "a7cb00dde55d424d9dc7ee1fc139c6dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36275 } }
I20260620 04:27:37.162698 14918 raft_consensus.cc:729] T c89d76c347dd49c1a4defb0c53c9a3fb P 126db552d7e645d597ecbab7867c8368 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 126db552d7e645d597ecbab7867c8368, State: Initialized, Role: FOLLOWER
I20260620 04:27:37.163228 14918 consensus_queue.cc:260] T c89d76c347dd49c1a4defb0c53c9a3fb P 126db552d7e645d597ecbab7867c8368 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0a07d8cbfd141e49fcce98f279a8ebc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36255 } } peers { permanent_uuid: "126db552d7e645d597ecbab7867c8368" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43567 } } peers { permanent_uuid: "a7cb00dde55d424d9dc7ee1fc139c6dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36275 } }
I20260620 04:27:37.164261 14918 ts_tablet_manager.cc:1428] T c89d76c347dd49c1a4defb0c53c9a3fb P 126db552d7e645d597ecbab7867c8368: Time spent starting tablet: real 0.005s user 0.008s sys 0.000s
Jun 20 04:27:37 dist-test-slave-nk6d krb5kdc[14118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929657, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:27:37.168766 14889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:37.156589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38715 (local address 127.0.98.62:36915)
0620 04:27:37.156735 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 04:27:37.156740 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:27:37.157347 (+ 607us) server_negotiation.cc:407] Connection header received
0620 04:27:37.158437 (+ 1090us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:27:37.158439 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:27:37.158488 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:27:37.158566 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:27:37.159979 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:37.160510 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:37.162068 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:37.162389 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:37.162488 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:27:37.165415 (+ 2927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:27:37.165419 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:27:37.165421 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:27:37.165450 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:27:37.166846 (+ 1396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:27:37.167330 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:27:37.167333 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:27:37.167334 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:27:37.167409 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:27:37.167717 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:27:37.167720 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:27:37.167722 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:27:37.167899 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:27:37.168011 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 04:27:37.168473 (+ 462us) server_negotiation.cc:299] Negotiation successful
0620 04:27:37.168606 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 04:27:37.169500 15029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:37.156697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36915 (local address 127.0.98.2:38715)
0620 04:27:37.157178 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:37.157229 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 04:27:37.158220 (+ 991us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:27:37.158727 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:27:37.158737 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:27:37.159174 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:27:37.159805 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:27:37.159824 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:27:37.160650 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:27:37.160654 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:27:37.161940 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:27:37.161950 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:27:37.162522 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:27:37.162526 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:27:37.162724 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:27:37.163672 (+ 948us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:27:37.163691 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:27:37.165247 (+ 1556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:27:37.166957 (+ 1710us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:27:37.166962 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:27:37.166977 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:27:37.167237 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:27:37.167488 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:27:37.167491 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:27:37.167493 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:27:37.167633 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:27:37.168011 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:27:37.168017 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:27:37.168255 (+ 238us) client_negotiation.cc:769] Sending connection context
0620 04:27:37.168416 (+ 161us) client_negotiation.cc:240] Negotiation successful
0620 04:27:37.168736 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":121,"threads_started":1}
I20260620 04:27:37.170563 15027 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36915
I20260620 04:27:37.170789 15027 heartbeater.cc:464] Registering TS with master...
I20260620 04:27:37.171257 15027 heartbeater.cc:511] Master 127.0.98.62:36915 requested a full tablet report, sending...
I20260620 04:27:37.172878 14699 ts_manager.cc:194] Registered new tserver with Master: 126db552d7e645d597ecbab7867c8368 (127.0.98.2:43567)
I20260620 04:27:37.173892 14699 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:38715
I20260620 04:27:37.176189 15027 heartbeater.cc:503] Master 127.0.98.62:36915 was elected leader, sending a full tablet report...
Jun 20 04:27:37 dist-test-slave-nk6d krb5kdc[14118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929656, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260620 04:27:37.183310 14894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:37.173673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43567 (local address 127.0.0.1:51694)
0620 04:27:37.173839 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:37.173853 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:27:37.173923 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:27:37.174880 (+ 957us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:27:37.174884 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:27:37.175070 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:27:37.175169 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:27:37.175176 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:27:37.176729 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:27:37.176733 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:27:37.177311 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:27:37.177319 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:27:37.178111 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:27:37.178125 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:27:37.178206 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:27:37.178665 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:27:37.178678 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:27:37.180017 (+ 1339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:27:37.181954 (+ 1937us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:27:37.181956 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:27:37.181958 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:27:37.182137 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:27:37.182416 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:27:37.182418 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:27:37.182419 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:27:37.182485 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:27:37.182912 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:27:37.182914 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:27:37.182985 (+ 71us) client_negotiation.cc:769] Sending connection context
0620 04:27:37.183084 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 04:27:37.183196 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96}
I20260620 04:27:37.183641 15032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:37.174171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51694 (local address 127.0.98.2:43567)
0620 04:27:37.174446 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0620 04:27:37.174458 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 04:27:37.174479 (+ 21us) server_negotiation.cc:407] Connection header received
0620 04:27:37.174545 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:27:37.174550 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:27:37.174748 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:27:37.174876 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:27:37.175410 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:37.176634 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:37.177416 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:37.177983 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:37.178123 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:27:37.180175 (+ 2052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:27:37.180184 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:27:37.180199 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:27:37.180222 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:27:37.181859 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:27:37.182225 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:27:37.182230 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:27:37.182233 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:27:37.182339 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:27:37.182564 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:27:37.182566 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:27:37.182568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:27:37.182830 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:27:37.182953 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 04:27:37.183171 (+ 218us) server_negotiation.cc:299] Negotiation successful
0620 04:27:37.183308 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":80,"threads_started":1}
I20260620 04:27:37.184625 14960 tablet_service.cc:1514] Processing DeleteTablet for tablet c89d76c347dd49c1a4defb0c53c9a3fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:27:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51694
I20260620 04:27:37.185492 15033 tablet_replica.cc:330] stopping tablet replica
I20260620 04:27:37.185758 15033 raft_consensus.cc:2229] T c89d76c347dd49c1a4defb0c53c9a3fb P 126db552d7e645d597ecbab7867c8368 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:27:37.185974 15033 raft_consensus.cc:2258] T c89d76c347dd49c1a4defb0c53c9a3fb P 126db552d7e645d597ecbab7867c8368 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:27:37.186900 15033 ts_tablet_manager.cc:1905] T c89d76c347dd49c1a4defb0c53c9a3fb P 126db552d7e645d597ecbab7867c8368: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:27:37.190343 15033 ts_tablet_manager.cc:1918] T c89d76c347dd49c1a4defb0c53c9a3fb P 126db552d7e645d597ecbab7867c8368: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:27:37.190423 15033 log.cc:1196] T c89d76c347dd49c1a4defb0c53c9a3fb P 126db552d7e645d597ecbab7867c8368: Deleting WAL directory at /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-1/wal/wals/c89d76c347dd49c1a4defb0c53c9a3fb
I20260620 04:27:37.190639 15033 ts_tablet_manager.cc:1939] T c89d76c347dd49c1a4defb0c53c9a3fb P 126db552d7e645d597ecbab7867c8368: Deleting consensus metadata
I20260620 04:27:37.192106 14685 catalog_manager.cc:4841] TS 126db552d7e645d597ecbab7867c8368 (127.0.98.2:43567): tablet c89d76c347dd49c1a4defb0c53c9a3fb (table test-table [id=77d4fd48ecec443ab6622073431c36c4]) successfully deleted
W20260620 04:27:37.275828 15030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:27:37.276125 15030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:27:37.276186 15030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:27:37.280145 15030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:27:37.280205 15030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:27:37.280282 15030 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 04:27:37.286348 15030 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:36275
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=42903
--tserver_master_addrs=127.0.98.62:36915
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15030
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:27:37.287362 15030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:27:37.288393 15030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:27:37.297503 15041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:27:37.297487 15038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:27:37.297492 15039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:27:37.297887 15030 server_base.cc:1034] running on GCE node
I20260620 04:27:37.298354 15030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:27:37.298980 15030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:27:37.300134 15030 hybrid_clock.cc:648] HybridClock initialized: now 1781929657300106 us; error 44 us; skew 500 ppm
Jun 20 04:27:37 dist-test-slave-nk6d krb5kdc[14118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929657, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:27:37.302776 15030 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 04:27:37.303702 15030 webserver.cc:460] Webserver started at http://127.0.98.3:42903/ using document root <none> and password file <none>
I20260620 04:27:37.304335 15030 fs_manager.cc:359] Metadata directory not provided
I20260620 04:27:37.304420 15030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:27:37.308342 15030 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 04:27:37.310488 15048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:27:37.311726 15030 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:27:37.311833 15030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-2/wal
uuid: "a7cb00dde55d424d9dc7ee1fc139c6dc"
format_stamp: "Formatted at 2026-06-20 04:27:36 on dist-test-slave-nk6d"
server_key: "dedf2077f899b792f7a0bb8539d31e1e"
server_key_iv: "f2b819fcd140ae7eed49935e59d790d3"
server_key_version: "encryptionkey@0"
I20260620 04:27:37.312397 15030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:27:37.330157 15030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:27:37.333215 15030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:27:37.333389 15030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:27:37.333966 15030 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:27:37.334762 15030 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:27:37.334806 15030 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:27:37.334858 15030 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:27:37.334887 15030 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:27:37.343374 15030 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36275
I20260620 04:27:37.343393 15161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36275 every 8 connection(s)
I20260620 04:27:37.344347 15030 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-2/data/info.pb
I20260620 04:27:37.348655 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 15030
Jun 20 04:27:37 dist-test-slave-nk6d krb5kdc[14118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929657, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:27:37.357002 15166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:37.349268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42262 (local address 127.0.98.62:36915)
0620 04:27:37.349490 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 04:27:37.349494 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:27:37.349574 (+ 80us) server_negotiation.cc:407] Connection header received
0620 04:27:37.349752 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:27:37.349758 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:27:37.349811 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:27:37.349902 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:27:37.350725 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:37.351516 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:37.352569 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:37.352874 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:37.352962 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:27:37.354160 (+ 1198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:27:37.354163 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:27:37.354165 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:27:37.354186 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:27:37.355524 (+ 1338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:27:37.355885 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:27:37.355889 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:27:37.355891 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:27:37.355989 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:27:37.356268 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:27:37.356270 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:27:37.356271 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:27:37.356428 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:27:37.356583 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 04:27:37.356754 (+ 171us) server_negotiation.cc:299] Negotiation successful
0620 04:27:37.356864 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":74,"threads_started":1}
I20260620 04:27:37.358100 14889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:37.346035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52443 (local address 127.0.98.62:36915)
0620 04:27:37.346196 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 04:27:37.346201 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:27:37.347031 (+ 830us) server_negotiation.cc:407] Connection header received
0620 04:27:37.348236 (+ 1205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:27:37.348239 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:27:37.348285 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:27:37.348358 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:27:37.349645 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:37.350348 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:37.351850 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:37.352222 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:37.352319 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:27:37.354790 (+ 2471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:27:37.354793 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:27:37.354794 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:27:37.354826 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:27:37.356097 (+ 1271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:27:37.356569 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:27:37.356572 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:27:37.356574 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:27:37.356647 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:27:37.357125 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:27:37.357128 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:27:37.357130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:27:37.357279 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:27:37.357366 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 04:27:37.357883 (+ 517us) server_negotiation.cc:299] Negotiation successful
0620 04:27:37.357993 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 04:27:37.359031 15164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:37.346426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36915 (local address 127.0.98.3:52443)
0620 04:27:37.346881 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:37.346929 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 04:27:37.348044 (+ 1115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:27:37.348517 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:27:37.348523 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:27:37.348934 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:27:37.349496 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:27:37.349516 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:27:37.350480 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:27:37.350483 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:27:37.351749 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:27:37.351758 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:27:37.352371 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:27:37.352374 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:27:37.352519 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:27:37.353095 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:27:37.353111 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:27:37.354631 (+ 1520us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:27:37.356190 (+ 1559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:27:37.356195 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:27:37.356210 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:27:37.356473 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:27:37.356732 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:27:37.356735 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:27:37.356737 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:27:37.356912 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:27:37.357376 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:27:37.357382 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:27:37.357644 (+ 262us) client_negotiation.cc:769] Sending connection context
0620 04:27:37.357831 (+ 187us) client_negotiation.cc:240] Negotiation successful
0620 04:27:37.358147 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":105,"threads_started":1}
I20260620 04:27:37.360038 15162 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36915
I20260620 04:27:37.360258 15162 heartbeater.cc:464] Registering TS with master...
I20260620 04:27:37.360707 15162 heartbeater.cc:511] Master 127.0.98.62:36915 requested a full tablet report, sending...
I20260620 04:27:37.361622 14699 ts_manager.cc:194] Registered new tserver with Master: a7cb00dde55d424d9dc7ee1fc139c6dc (127.0.98.3:36275)
I20260620 04:27:37.362139 14699 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52443
I20260620 04:27:37.370384 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 04:27:37.374511 14889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:37.371000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42264 (local address 127.0.98.62:36915)
0620 04:27:37.371118 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 04:27:37.371122 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:27:37.371303 (+ 181us) server_negotiation.cc:407] Connection header received
0620 04:27:37.371446 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:27:37.371448 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:27:37.371555 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:27:37.371638 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:27:37.371826 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:37.372404 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:37.373234 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:37.373445 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:37.373522 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:27:37.373800 (+ 278us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:27:37.374204 (+ 404us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:27:37.374291 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 04:27:37.374368 (+ 77us) server_negotiation.cc:299] Negotiation successful
0620 04:27:37.374411 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 04:27:37.375133 14699 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:42264:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 04:27:37.378454 14699 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 04:27:37.385527 14821 tablet_service.cc:1467] Processing CreateTablet for tablet fd554729413d4b398978509758144e49 (DEFAULT_TABLE table=test-table [id=aca8f54962f347509c28db3f0497c855]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:27:37.385527 14960 tablet_service.cc:1467] Processing CreateTablet for tablet fd554729413d4b398978509758144e49 (DEFAULT_TABLE table=test-table [id=aca8f54962f347509c28db3f0497c855]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:27:37.386157 14960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd554729413d4b398978509758144e49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:27:37.386168 14821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd554729413d4b398978509758144e49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:27:37.388877 14918 tablet_bootstrap.cc:492] T fd554729413d4b398978509758144e49 P 126db552d7e645d597ecbab7867c8368: Bootstrap starting.
I20260620 04:27:37.388877 14779 tablet_bootstrap.cc:492] T fd554729413d4b398978509758144e49 P a0a07d8cbfd141e49fcce98f279a8ebc: Bootstrap starting.
I20260620 04:27:37.389246 14894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:37.384721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36275 (local address 127.0.0.1:37858)
0620 04:27:37.384818 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:37.384833 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 04:27:37.384909 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:27:37.385838 (+ 929us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:27:37.385841 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:27:37.385861 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:27:37.385946 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:27:37.385951 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:27:37.387173 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:27:37.387177 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:27:37.387935 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:27:37.387943 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:27:37.388836 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:27:37.388839 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:27:37.388925 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:27:37.388965 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 04:27:37.389036 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 04:27:37.389120 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260620 04:27:37.389550 15171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:37.385075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37858 (local address 127.0.98.3:36275)
0620 04:27:37.385408 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0620 04:27:37.385418 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 04:27:37.385439 (+ 21us) server_negotiation.cc:407] Connection header received
0620 04:27:37.385502 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:27:37.385506 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:27:37.385681 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:27:37.386084 (+ 403us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:27:37.386219 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:37.387048 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:37.388097 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:37.388720 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:37.388818 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:27:37.389085 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0620 04:27:37.389138 (+ 53us) server_negotiation.cc:299] Negotiation successful
0620 04:27:37.389227 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":143,"threads_started":1}
I20260620 04:27:37.389700 14779 tablet_bootstrap.cc:654] T fd554729413d4b398978509758144e49 P a0a07d8cbfd141e49fcce98f279a8ebc: Neither blocks nor log segments found. Creating new log.
I20260620 04:27:37.389740 14918 tablet_bootstrap.cc:654] T fd554729413d4b398978509758144e49 P 126db552d7e645d597ecbab7867c8368: Neither blocks nor log segments found. Creating new log.
I20260620 04:27:37.391403 14779 tablet_bootstrap.cc:492] T fd554729413d4b398978509758144e49 P a0a07d8cbfd141e49fcce98f279a8ebc: No bootstrap required, opened a new log
I20260620 04:27:37.391448 14918 tablet_bootstrap.cc:492] T fd554729413d4b398978509758144e49 P 126db552d7e645d597ecbab7867c8368: No bootstrap required, opened a new log
I20260620 04:27:37.391487 14779 ts_tablet_manager.cc:1397] T fd554729413d4b398978509758144e49 P a0a07d8cbfd141e49fcce98f279a8ebc: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 04:27:37.391534 14918 ts_tablet_manager.cc:1397] T fd554729413d4b398978509758144e49 P 126db552d7e645d597ecbab7867c8368: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 04:27:37.391881 14779 raft_consensus.cc:348] T fd554729413d4b398978509758144e49 P a0a07d8cbfd141e49fcce98f279a8ebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7cb00dde55d424d9dc7ee1fc139c6dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36275 } } peers { permanent_uuid: "a0a07d8cbfd141e49fcce98f279a8ebc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36255 } } peers { permanent_uuid: "126db552d7e645d597ecbab7867c8368" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43567 } }
I20260620 04:27:37.392006 14779 raft_consensus.cc:374] T fd554729413d4b398978509758144e49 P a0a07d8cbfd141e49fcce98f279a8ebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:27:37.391950 14918 raft_consensus.cc:348] T fd554729413d4b398978509758144e49 P 126db552d7e645d597ecbab7867c8368 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7cb00dde55d424d9dc7ee1fc139c6dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36275 } } peers { permanent_uuid: "a0a07d8cbfd141e49fcce98f279a8ebc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36255 } } peers { permanent_uuid: "126db552d7e645d597ecbab7867c8368" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43567 } }
I20260620 04:27:37.392050 14779 raft_consensus.cc:729] T fd554729413d4b398978509758144e49 P a0a07d8cbfd141e49fcce98f279a8ebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0a07d8cbfd141e49fcce98f279a8ebc, State: Initialized, Role: FOLLOWER
I20260620 04:27:37.392086 14918 raft_consensus.cc:374] T fd554729413d4b398978509758144e49 P 126db552d7e645d597ecbab7867c8368 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:27:37.392119 14918 raft_consensus.cc:729] T fd554729413d4b398978509758144e49 P 126db552d7e645d597ecbab7867c8368 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 126db552d7e645d597ecbab7867c8368, State: Initialized, Role: FOLLOWER
I20260620 04:27:37.392165 14779 consensus_queue.cc:260] T fd554729413d4b398978509758144e49 P a0a07d8cbfd141e49fcce98f279a8ebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7cb00dde55d424d9dc7ee1fc139c6dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36275 } } peers { permanent_uuid: "a0a07d8cbfd141e49fcce98f279a8ebc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36255 } } peers { permanent_uuid: "126db552d7e645d597ecbab7867c8368" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43567 } }
I20260620 04:27:37.392230 14918 consensus_queue.cc:260] T fd554729413d4b398978509758144e49 P 126db552d7e645d597ecbab7867c8368 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7cb00dde55d424d9dc7ee1fc139c6dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36275 } } peers { permanent_uuid: "a0a07d8cbfd141e49fcce98f279a8ebc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36255 } } peers { permanent_uuid: "126db552d7e645d597ecbab7867c8368" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43567 } }
I20260620 04:27:37.392484 14918 ts_tablet_manager.cc:1428] T fd554729413d4b398978509758144e49 P 126db552d7e645d597ecbab7867c8368: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 04:27:37.392537 14779 ts_tablet_manager.cc:1428] T fd554729413d4b398978509758144e49 P a0a07d8cbfd141e49fcce98f279a8ebc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 04:27:37.392556 15096 tablet_service.cc:1467] Processing CreateTablet for tablet fd554729413d4b398978509758144e49 (DEFAULT_TABLE table=test-table [id=aca8f54962f347509c28db3f0497c855]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:27:37.393591 15096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd554729413d4b398978509758144e49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:27:37.400259 15174 tablet_bootstrap.cc:492] T fd554729413d4b398978509758144e49 P a7cb00dde55d424d9dc7ee1fc139c6dc: Bootstrap starting.
I20260620 04:27:37.402274 15174 tablet_bootstrap.cc:654] T fd554729413d4b398978509758144e49 P a7cb00dde55d424d9dc7ee1fc139c6dc: Neither blocks nor log segments found. Creating new log.
I20260620 04:27:37.403326 15174 log.cc:824] T fd554729413d4b398978509758144e49 P a7cb00dde55d424d9dc7ee1fc139c6dc: Log is configured to *not* fsync() on all Append() calls
I20260620 04:27:37.405085 15174 tablet_bootstrap.cc:492] T fd554729413d4b398978509758144e49 P a7cb00dde55d424d9dc7ee1fc139c6dc: No bootstrap required, opened a new log
I20260620 04:27:37.405261 15174 ts_tablet_manager.cc:1397] T fd554729413d4b398978509758144e49 P a7cb00dde55d424d9dc7ee1fc139c6dc: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
W20260620 04:27:37.405742 15028 tablet.cc:2369] T fd554729413d4b398978509758144e49 P 126db552d7e645d597ecbab7867c8368: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 04:27:37.408353 15174 raft_consensus.cc:348] T fd554729413d4b398978509758144e49 P a7cb00dde55d424d9dc7ee1fc139c6dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7cb00dde55d424d9dc7ee1fc139c6dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36275 } } peers { permanent_uuid: "a0a07d8cbfd141e49fcce98f279a8ebc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36255 } } peers { permanent_uuid: "126db552d7e645d597ecbab7867c8368" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43567 } }
I20260620 04:27:37.408586 15174 raft_consensus.cc:374] T fd554729413d4b398978509758144e49 P a7cb00dde55d424d9dc7ee1fc139c6dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:27:37.408641 15174 raft_consensus.cc:729] T fd554729413d4b398978509758144e49 P a7cb00dde55d424d9dc7ee1fc139c6dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7cb00dde55d424d9dc7ee1fc139c6dc, State: Initialized, Role: FOLLOWER
I20260620 04:27:37.409076 15174 consensus_queue.cc:260] T fd554729413d4b398978509758144e49 P a7cb00dde55d424d9dc7ee1fc139c6dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7cb00dde55d424d9dc7ee1fc139c6dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36275 } } peers { permanent_uuid: "a0a07d8cbfd141e49fcce98f279a8ebc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36255 } } peers { permanent_uuid: "126db552d7e645d597ecbab7867c8368" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43567 } }
I20260620 04:27:37.409673 15162 heartbeater.cc:503] Master 127.0.98.62:36915 was elected leader, sending a full tablet report...
I20260620 04:27:37.409904 15174 ts_tablet_manager.cc:1428] T fd554729413d4b398978509758144e49 P a7cb00dde55d424d9dc7ee1fc139c6dc: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 04:27:37.437532 15176 raft_consensus.cc:482] T fd554729413d4b398978509758144e49 P a7cb00dde55d424d9dc7ee1fc139c6dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:27:37.437743 15176 raft_consensus.cc:504] T fd554729413d4b398978509758144e49 P a7cb00dde55d424d9dc7ee1fc139c6dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7cb00dde55d424d9dc7ee1fc139c6dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36275 } } peers { permanent_uuid: "a0a07d8cbfd141e49fcce98f279a8ebc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36255 } } peers { permanent_uuid: "126db552d7e645d597ecbab7867c8368" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43567 } }
I20260620 04:27:37.438882 15176 leader_election.cc:288] T fd554729413d4b398978509758144e49 P a7cb00dde55d424d9dc7ee1fc139c6dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0a07d8cbfd141e49fcce98f279a8ebc (127.0.98.1:36255), 126db552d7e645d597ecbab7867c8368 (127.0.98.2:43567)
I20260620 04:27:37.442827 14895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:37.439232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45005 (local address 127.0.98.1:36255)
0620 04:27:37.439353 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 04:27:37.439357 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:27:37.439675 (+ 318us) server_negotiation.cc:407] Connection header received
0620 04:27:37.439765 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:27:37.439768 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:27:37.439829 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:27:37.439934 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:27:37.440200 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:37.440785 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:37.441776 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:37.442321 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:37.442417 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:27:37.442577 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 04:27:37.442632 (+ 55us) server_negotiation.cc:299] Negotiation successful
0620 04:27:37.442703 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 04:27:37.442853 15177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:37.439148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36255 (local address 127.0.98.3:45005)
0620 04:27:37.439562 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:37.439575 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:27:37.439680 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:27:37.439940 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:27:37.439942 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:27:37.439977 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:27:37.440090 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:27:37.440096 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:27:37.440880 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:27:37.440883 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:27:37.441664 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:27:37.441674 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:27:37.442420 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:27:37.442423 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:27:37.442515 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:27:37.442546 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 04:27:37.442620 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 04:27:37.442689 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"mutex_wait_us":35,"thread_start_us":111,"threads_started":1}
I20260620 04:27:37.443123 15032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:37.439311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49207 (local address 127.0.98.2:43567)
0620 04:27:37.439416 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0620 04:27:37.439420 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:27:37.439435 (+ 15us) server_negotiation.cc:407] Connection header received
0620 04:27:37.439551 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:27:37.439555 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:27:37.439623 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:27:37.439726 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:27:37.440073 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:37.440617 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:37.441705 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:37.442304 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:37.442403 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:27:37.442568 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0620 04:27:37.442927 (+ 359us) server_negotiation.cc:299] Negotiation successful
0620 04:27:37.443005 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 04:27:37.443380 15164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:37.439148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43567 (local address 127.0.98.3:49207)
0620 04:27:37.439336 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:37.439351 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 04:27:37.439463 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:27:37.439763 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:27:37.439766 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:27:37.439786 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:27:37.439909 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:27:37.439916 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:27:37.440723 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:27:37.440727 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:27:37.441579 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:27:37.441588 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:27:37.442636 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:27:37.442639 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:27:37.442735 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:27:37.442754 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 04:27:37.442812 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 04:27:37.442869 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114}
I20260620 04:27:37.443557 14841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd554729413d4b398978509758144e49" candidate_uuid: "a7cb00dde55d424d9dc7ee1fc139c6dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0a07d8cbfd141e49fcce98f279a8ebc" is_pre_election: true
I20260620 04:27:37.443884 14841 raft_consensus.cc:2454] T fd554729413d4b398978509758144e49 P a0a07d8cbfd141e49fcce98f279a8ebc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7cb00dde55d424d9dc7ee1fc139c6dc in term 0.
I20260620 04:27:37.443912 14980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd554729413d4b398978509758144e49" candidate_uuid: "a7cb00dde55d424d9dc7ee1fc139c6dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "126db552d7e645d597ecbab7867c8368" is_pre_election: true
I20260620 04:27:37.444232 14980 raft_consensus.cc:2454] T fd554729413d4b398978509758144e49 P 126db552d7e645d597ecbab7867c8368 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7cb00dde55d424d9dc7ee1fc139c6dc in term 0.
I20260620 04:27:37.444370 15050 leader_election.cc:302] T fd554729413d4b398978509758144e49 P a7cb00dde55d424d9dc7ee1fc139c6dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0a07d8cbfd141e49fcce98f279a8ebc, a7cb00dde55d424d9dc7ee1fc139c6dc; no voters:
I20260620 04:27:37.444662 15176 raft_consensus.cc:2781] T fd554729413d4b398978509758144e49 P a7cb00dde55d424d9dc7ee1fc139c6dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 04:27:37.444741 15176 raft_consensus.cc:482] T fd554729413d4b398978509758144e49 P a7cb00dde55d424d9dc7ee1fc139c6dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 04:27:37.444770 15176 raft_consensus.cc:3037] T fd554729413d4b398978509758144e49 P a7cb00dde55d424d9dc7ee1fc139c6dc [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:27:37.445740 15176 raft_consensus.cc:504] T fd554729413d4b398978509758144e49 P a7cb00dde55d424d9dc7ee1fc139c6dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7cb00dde55d424d9dc7ee1fc139c6dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36275 } } peers { permanent_uuid: "a0a07d8cbfd141e49fcce98f279a8ebc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36255 } } peers { permanent_uuid: "126db552d7e645d597ecbab7867c8368" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43567 } }
I20260620 04:27:37.446166 15176 leader_election.cc:288] T fd554729413d4b398978509758144e49 P a7cb00dde55d424d9dc7ee1fc139c6dc [CANDIDATE]: Term 1 election: Requested vote from peers a0a07d8cbfd141e49fcce98f279a8ebc (127.0.98.1:36255), 126db552d7e645d597ecbab7867c8368 (127.0.98.2:43567)
I20260620 04:27:37.446564 14841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd554729413d4b398978509758144e49" candidate_uuid: "a7cb00dde55d424d9dc7ee1fc139c6dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0a07d8cbfd141e49fcce98f279a8ebc"
I20260620 04:27:37.446697 14841 raft_consensus.cc:3037] T fd554729413d4b398978509758144e49 P a0a07d8cbfd141e49fcce98f279a8ebc [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:27:37.446707 14980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd554729413d4b398978509758144e49" candidate_uuid: "a7cb00dde55d424d9dc7ee1fc139c6dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "126db552d7e645d597ecbab7867c8368"
I20260620 04:27:37.446825 14980 raft_consensus.cc:3037] T fd554729413d4b398978509758144e49 P 126db552d7e645d597ecbab7867c8368 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:27:37.447768 14841 raft_consensus.cc:2454] T fd554729413d4b398978509758144e49 P a0a07d8cbfd141e49fcce98f279a8ebc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7cb00dde55d424d9dc7ee1fc139c6dc in term 1.
I20260620 04:27:37.447788 14980 raft_consensus.cc:2454] T fd554729413d4b398978509758144e49 P 126db552d7e645d597ecbab7867c8368 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7cb00dde55d424d9dc7ee1fc139c6dc in term 1.
I20260620 04:27:37.448040 15050 leader_election.cc:302] T fd554729413d4b398978509758144e49 P a7cb00dde55d424d9dc7ee1fc139c6dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0a07d8cbfd141e49fcce98f279a8ebc, a7cb00dde55d424d9dc7ee1fc139c6dc; no voters:
I20260620 04:27:37.448212 15176 raft_consensus.cc:2781] T fd554729413d4b398978509758144e49 P a7cb00dde55d424d9dc7ee1fc139c6dc [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:27:37.448418 15176 raft_consensus.cc:686] T fd554729413d4b398978509758144e49 P a7cb00dde55d424d9dc7ee1fc139c6dc [term 1 LEADER]: Becoming Leader. State: Replica: a7cb00dde55d424d9dc7ee1fc139c6dc, State: Running, Role: LEADER
I20260620 04:27:37.448748 15176 consensus_queue.cc:237] T fd554729413d4b398978509758144e49 P a7cb00dde55d424d9dc7ee1fc139c6dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7cb00dde55d424d9dc7ee1fc139c6dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36275 } } peers { permanent_uuid: "a0a07d8cbfd141e49fcce98f279a8ebc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36255 } } peers { permanent_uuid: "126db552d7e645d597ecbab7867c8368" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43567 } }
I20260620 04:27:37.451709 14698 catalog_manager.cc:5495] T fd554729413d4b398978509758144e49 P a7cb00dde55d424d9dc7ee1fc139c6dc reported cstate change: term changed from 0 to 1, leader changed from <none> to a7cb00dde55d424d9dc7ee1fc139c6dc (127.0.98.3). New cstate: current_term: 1 leader_uuid: "a7cb00dde55d424d9dc7ee1fc139c6dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7cb00dde55d424d9dc7ee1fc139c6dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36275 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a0a07d8cbfd141e49fcce98f279a8ebc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36255 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "126db552d7e645d597ecbab7867c8368" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43567 } health_report { overall_health: UNKNOWN } } }
W20260620 04:27:37.460337 14888 tablet.cc:2369] T fd554729413d4b398978509758144e49 P a0a07d8cbfd141e49fcce98f279a8ebc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 04:27:37.463244 15171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:37.459827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37860 (local address 127.0.98.3:36275)
0620 04:27:37.459944 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 04:27:37.459947 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:27:37.459979 (+ 32us) server_negotiation.cc:407] Connection header received
0620 04:27:37.460153 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:27:37.460171 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:27:37.460225 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:27:37.460299 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:27:37.460525 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:37.461039 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:37.461868 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:37.462114 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:37.462194 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:27:37.462449 (+ 255us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:27:37.462865 (+ 416us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:27:37.462984 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 04:27:37.463037 (+ 53us) server_negotiation.cc:299] Negotiation successful
0620 04:27:37.463087 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 04:27:37.468842 14980 raft_consensus.cc:1261] T fd554729413d4b398978509758144e49 P 126db552d7e645d597ecbab7867c8368 [term 1 FOLLOWER]: Refusing update from remote peer a7cb00dde55d424d9dc7ee1fc139c6dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:27:37.468842 14841 raft_consensus.cc:1261] T fd554729413d4b398978509758144e49 P a0a07d8cbfd141e49fcce98f279a8ebc [term 1 FOLLOWER]: Refusing update from remote peer a7cb00dde55d424d9dc7ee1fc139c6dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:27:37.469473 15178 consensus_queue.cc:1035] T fd554729413d4b398978509758144e49 P a7cb00dde55d424d9dc7ee1fc139c6dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "126db552d7e645d597ecbab7867c8368" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:27:37.469638 15176 consensus_queue.cc:1035] T fd554729413d4b398978509758144e49 P a7cb00dde55d424d9dc7ee1fc139c6dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0a07d8cbfd141e49fcce98f279a8ebc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:27:37.477883 15182 mvcc.cc:205] Tried to move back new op lower bound from 7298783876985622528 to 7298783876912644096. Current Snapshot: MvccSnapshot[applied={T|T < 7298783876985622528}]
I20260620 04:27:37.483306 14699 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:42264:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:27:37.483459 14699 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:42264:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:27:37.485531 14699 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a87a20f134ed4ae9a447a40942f3d7ea: Sending DeleteTablet for 3 replicas of tablet fd554729413d4b398978509758144e49
I20260620 04:27:37.486222 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 14737
I20260620 04:27:37.486418 15096 tablet_service.cc:1514] Processing DeleteTablet for tablet fd554729413d4b398978509758144e49 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:27:37 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37858
I20260620 04:27:37.486778 14960 tablet_service.cc:1514] Processing DeleteTablet for tablet fd554729413d4b398978509758144e49 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:27:37 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51694
I20260620 04:27:37.487914 15033 tablet_replica.cc:330] T fd554729413d4b398978509758144e49 P 126db552d7e645d597ecbab7867c8368: stopping tablet replica
I20260620 04:27:37.488054 15033 raft_consensus.cc:2229] T fd554729413d4b398978509758144e49 P 126db552d7e645d597ecbab7867c8368 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:27:37.488193 15033 raft_consensus.cc:2258] T fd554729413d4b398978509758144e49 P 126db552d7e645d597ecbab7867c8368 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:27:37.489935 15033 ts_tablet_manager.cc:1905] T fd554729413d4b398978509758144e49 P 126db552d7e645d597ecbab7867c8368: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:27:37.492059 15190 tablet_replica.cc:330] T fd554729413d4b398978509758144e49 P a7cb00dde55d424d9dc7ee1fc139c6dc: stopping tablet replica
I20260620 04:27:37.492350 15033 ts_tablet_manager.cc:1918] T fd554729413d4b398978509758144e49 P 126db552d7e645d597ecbab7867c8368: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:27:37.492435 15033 log.cc:1196] T fd554729413d4b398978509758144e49 P 126db552d7e645d597ecbab7867c8368: Deleting WAL directory at /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-1/wal/wals/fd554729413d4b398978509758144e49
I20260620 04:27:37.492548 15190 raft_consensus.cc:2229] T fd554729413d4b398978509758144e49 P a7cb00dde55d424d9dc7ee1fc139c6dc [term 1 LEADER]: Raft consensus shutting down.
I20260620 04:27:37.492657 15033 ts_tablet_manager.cc:1939] T fd554729413d4b398978509758144e49 P 126db552d7e645d597ecbab7867c8368: Deleting consensus metadata
I20260620 04:27:37.493041 15190 raft_consensus.cc:2258] T fd554729413d4b398978509758144e49 P a7cb00dde55d424d9dc7ee1fc139c6dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:27:37.493376 14685 catalog_manager.cc:4841] TS 126db552d7e645d597ecbab7867c8368 (127.0.98.2:43567): tablet fd554729413d4b398978509758144e49 (table test-table [id=aca8f54962f347509c28db3f0497c855]) successfully deleted
W20260620 04:27:37.493599 14684 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36255 (error 108)
I20260620 04:27:37.494609 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 14891
I20260620 04:27:37.494935 14894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:37.494627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36255 (local address 127.0.0.1:54980)
0620 04:27:37.494795 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:37.494856 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260620 04:27:37.495311 15190 ts_tablet_manager.cc:1905] T fd554729413d4b398978509758144e49 P a7cb00dde55d424d9dc7ee1fc139c6dc: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 04:27:37.495282 14684 catalog_manager.cc:4584] TS a0a07d8cbfd141e49fcce98f279a8ebc (127.0.98.1:36255): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fd554729413d4b398978509758144e49: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36255: connect: Connection refused (error 111)
I20260620 04:27:37.498055 15190 ts_tablet_manager.cc:1918] T fd554729413d4b398978509758144e49 P a7cb00dde55d424d9dc7ee1fc139c6dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:27:37.498157 15190 log.cc:1196] T fd554729413d4b398978509758144e49 P a7cb00dde55d424d9dc7ee1fc139c6dc: Deleting WAL directory at /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-2/wal/wals/fd554729413d4b398978509758144e49
I20260620 04:27:37.498500 15190 ts_tablet_manager.cc:1939] T fd554729413d4b398978509758144e49 P a7cb00dde55d424d9dc7ee1fc139c6dc: Deleting consensus metadata
I20260620 04:27:37.499776 14684 catalog_manager.cc:4841] TS a7cb00dde55d424d9dc7ee1fc139c6dc (127.0.98.3:36275): tablet fd554729413d4b398978509758144e49 (table test-table [id=aca8f54962f347509c28db3f0497c855]) successfully deleted
I20260620 04:27:37.500438 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 15030
I20260620 04:27:37.505484 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 14666
I20260620 04:27:37.510911 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36915
--webserver_interface=127.0.98.62
--webserver_port=35683
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36915
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:27:37.620723 15192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:27:37.620998 15192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:27:37.621037 15192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:27:37.625087 15192 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:27:37.625159 15192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:27:37.625178 15192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:27:37.625193 15192 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:27:37.625207 15192 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:27:37.630530 15192 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36915
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36915
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=35683
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15192
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:27:37.631575 15192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:27:37.632642 15192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:27:37.640650 15197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:27:37.640842 15192 server_base.cc:1034] running on GCE node
W20260620 04:27:37.640662 15200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:27:37.640662 15198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:27:37.641553 15192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:27:37.642625 15192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:27:37.643782 15192 hybrid_clock.cc:648] HybridClock initialized: now 1781929657643764 us; error 32 us; skew 500 ppm
Jun 20 04:27:37 dist-test-slave-nk6d krb5kdc[14118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929657, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:27:37.646353 15192 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 04:27:37.647245 15192 webserver.cc:460] Webserver started at http://127.0.98.62:35683/ using document root <none> and password file <none>
I20260620 04:27:37.647746 15192 fs_manager.cc:359] Metadata directory not provided
I20260620 04:27:37.647801 15192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:27:37.651475 15192 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:27:37.653501 15207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:27:37.654722 15192 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 04:27:37.654819 15192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/master-0/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/master-0/wal
uuid: "a87a20f134ed4ae9a447a40942f3d7ea"
format_stamp: "Formatted at 2026-06-20 04:27:35 on dist-test-slave-nk6d"
server_key: "812718ef149fdf619ec957fc50632200"
server_key_iv: "1502d5d4f62daadbaabd2bdeee1d2496"
server_key_version: "encryptionkey@0"
I20260620 04:27:37.655336 15192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:27:37.664660 15192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:27:37.667224 15192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:27:37.667366 15192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:27:37.673825 15192 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36915
I20260620 04:27:37.674069 15259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36915 every 8 connection(s)
I20260620 04:27:37.674881 15192 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:27:37.676699 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 15192
I20260620 04:27:37.677073 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:36255
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:36915
--webserver_port=36093
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 04:27:37.679374 15260 sys_catalog.cc:263] Verifying existing consensus state
I20260620 04:27:37.681501 15260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a87a20f134ed4ae9a447a40942f3d7ea: Bootstrap starting.
I20260620 04:27:37.704222 15260 log.cc:824] T 00000000000000000000000000000000 P a87a20f134ed4ae9a447a40942f3d7ea: Log is configured to *not* fsync() on all Append() calls
I20260620 04:27:37.717849 15260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a87a20f134ed4ae9a447a40942f3d7ea: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 04:27:37.718292 15260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a87a20f134ed4ae9a447a40942f3d7ea: Bootstrap complete.
I20260620 04:27:37.723562 15260 raft_consensus.cc:348] T 00000000000000000000000000000000 P a87a20f134ed4ae9a447a40942f3d7ea [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a87a20f134ed4ae9a447a40942f3d7ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36915 } }
I20260620 04:27:37.724162 15260 raft_consensus.cc:729] T 00000000000000000000000000000000 P a87a20f134ed4ae9a447a40942f3d7ea [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: a87a20f134ed4ae9a447a40942f3d7ea, State: Initialized, Role: FOLLOWER
I20260620 04:27:37.724721 15260 consensus_queue.cc:260] T 00000000000000000000000000000000 P a87a20f134ed4ae9a447a40942f3d7ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a87a20f134ed4ae9a447a40942f3d7ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36915 } }
I20260620 04:27:37.724845 15260 raft_consensus.cc:388] T 00000000000000000000000000000000 P a87a20f134ed4ae9a447a40942f3d7ea [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:27:37.724911 15260 raft_consensus.cc:482] T 00000000000000000000000000000000 P a87a20f134ed4ae9a447a40942f3d7ea [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:27:37.725054 15260 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a87a20f134ed4ae9a447a40942f3d7ea [term 2 FOLLOWER]: Advancing to term 3
I20260620 04:27:37.726955 15260 raft_consensus.cc:504] T 00000000000000000000000000000000 P a87a20f134ed4ae9a447a40942f3d7ea [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a87a20f134ed4ae9a447a40942f3d7ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36915 } }
I20260620 04:27:37.727267 15260 leader_election.cc:302] T 00000000000000000000000000000000 P a87a20f134ed4ae9a447a40942f3d7ea [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a87a20f134ed4ae9a447a40942f3d7ea; no voters:
I20260620 04:27:37.727550 15260 leader_election.cc:288] T 00000000000000000000000000000000 P a87a20f134ed4ae9a447a40942f3d7ea [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 04:27:37.727658 15264 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a87a20f134ed4ae9a447a40942f3d7ea [term 3 FOLLOWER]: Leader election won for term 3
I20260620 04:27:37.727846 15264 raft_consensus.cc:686] T 00000000000000000000000000000000 P a87a20f134ed4ae9a447a40942f3d7ea [term 3 LEADER]: Becoming Leader. State: Replica: a87a20f134ed4ae9a447a40942f3d7ea, State: Running, Role: LEADER
I20260620 04:27:37.728163 15264 consensus_queue.cc:237] T 00000000000000000000000000000000 P a87a20f134ed4ae9a447a40942f3d7ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a87a20f134ed4ae9a447a40942f3d7ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36915 } }
I20260620 04:27:37.728840 15260 sys_catalog.cc:564] T 00000000000000000000000000000000 P a87a20f134ed4ae9a447a40942f3d7ea [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:27:37.729095 15264 sys_catalog.cc:455] T 00000000000000000000000000000000 P a87a20f134ed4ae9a447a40942f3d7ea [sys.catalog]: SysCatalogTable state changed. Reason: New leader a87a20f134ed4ae9a447a40942f3d7ea. Latest consensus state: current_term: 3 leader_uuid: "a87a20f134ed4ae9a447a40942f3d7ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a87a20f134ed4ae9a447a40942f3d7ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36915 } } }
I20260620 04:27:37.729203 15264 sys_catalog.cc:458] T 00000000000000000000000000000000 P a87a20f134ed4ae9a447a40942f3d7ea [sys.catalog]: This master's current role is: LEADER
I20260620 04:27:37.729552 15268 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:27:37.729584 15265 sys_catalog.cc:455] T 00000000000000000000000000000000 P a87a20f134ed4ae9a447a40942f3d7ea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "a87a20f134ed4ae9a447a40942f3d7ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a87a20f134ed4ae9a447a40942f3d7ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36915 } } }
I20260620 04:27:37.729781 15265 sys_catalog.cc:458] T 00000000000000000000000000000000 P a87a20f134ed4ae9a447a40942f3d7ea [sys.catalog]: This master's current role is: LEADER
I20260620 04:27:37.734640 15268 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:27:37.735585 15268 catalog_manager.cc:1259] Loaded cluster ID: 676c7b67d90c4e8a855460a0f74a3aa4
I20260620 04:27:37.735651 15268 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260620 04:27:37.736688 15268 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fa94be7c9ad google::LogMessage::Fail() at ??:0
@ 0x7fa94be7f88a google::LogMessage::SendToLog() at ??:0
@ 0x7fa94be7c4de google::LogMessage::Flush() at ??:0
@ 0x7fa94be7da19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fa9567bbfdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fa9567bcd2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fa9567b9579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fa9567fd640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fa94e0ce3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fa94e0cec83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fa94e0d03f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fa94e0bf512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fa94f7036db start_thread at ??:0
@ 0x7fa94a97a61f clone at ??:0
W20260620 04:27:37.829262 15262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:27:37.829620 15262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:27:37.829689 15262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:27:37.835753 15262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:27:37.835846 15262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:27:37.835937 15262 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 04:27:37.844081 15262 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:36255
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=36093
--tserver_master_addrs=127.0.98.62:36915
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15262
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:27:37.845541 15262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:27:37.846884 15262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:27:37.857854 15301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:27:37.857918 15302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:27:37.858070 15262 server_base.cc:1034] running on GCE node
W20260620 04:27:37.857854 15304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:27:37.858574 15262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:27:37.859223 15262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:27:37.860402 15262 hybrid_clock.cc:648] HybridClock initialized: now 1781929657860366 us; error 53 us; skew 500 ppm
Jun 20 04:27:37 dist-test-slave-nk6d krb5kdc[14118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929657, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:27:37.870036 15262 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 04:27:37.871287 15262 webserver.cc:460] Webserver started at http://127.0.98.1:36093/ using document root <none> and password file <none>
I20260620 04:27:37.872066 15262 fs_manager.cc:359] Metadata directory not provided
I20260620 04:27:37.872174 15262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:27:37.877709 15262 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 04:27:37.880627 15311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:27:37.882200 15262 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 04:27:37.882344 15262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-0/wal
uuid: "a0a07d8cbfd141e49fcce98f279a8ebc"
format_stamp: "Formatted at 2026-06-20 04:27:35 on dist-test-slave-nk6d"
server_key: "a118e53c149d76aa58ea1913d93a045b"
server_key_iv: "536eba379b58ca4676c768769cb42be3"
server_key_version: "encryptionkey@0"
I20260620 04:27:37.883112 15262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:27:37.904814 15262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:27:37.908767 15262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:27:37.908969 15262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:27:37.909683 15262 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:27:37.910849 15318 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 04:27:37.914970 15262 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 04:27:37.915050 15262 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260620 04:27:37.915115 15262 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 04:27:37.917234 15262 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 04:27:37.917279 15262 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260620 04:27:37.917636 15318 tablet_bootstrap.cc:492] T fd554729413d4b398978509758144e49 P a0a07d8cbfd141e49fcce98f279a8ebc: Bootstrap starting.
I20260620 04:27:37.927172 15262 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36255
I20260620 04:27:37.927204 15425 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36255 every 8 connection(s)
I20260620 04:27:37.928354 15262 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-0/data/info.pb
I20260620 04:27:37.934687 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 15262
I20260620 04:27:37.935130 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:43567
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:36915
--webserver_port=43001
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 04:27:37.939805 15318 log.cc:824] T fd554729413d4b398978509758144e49 P a0a07d8cbfd141e49fcce98f279a8ebc: Log is configured to *not* fsync() on all Append() calls
I20260620 04:27:37.949299 15318 tablet_bootstrap.cc:492] T fd554729413d4b398978509758144e49 P a0a07d8cbfd141e49fcce98f279a8ebc: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 04:27:37.949753 15318 tablet_bootstrap.cc:492] T fd554729413d4b398978509758144e49 P a0a07d8cbfd141e49fcce98f279a8ebc: Bootstrap complete.
I20260620 04:27:37.951082 15318 ts_tablet_manager.cc:1397] T fd554729413d4b398978509758144e49 P a0a07d8cbfd141e49fcce98f279a8ebc: Time spent bootstrapping tablet: real 0.034s user 0.026s sys 0.004s
I20260620 04:27:37.954485 15318 raft_consensus.cc:348] T fd554729413d4b398978509758144e49 P a0a07d8cbfd141e49fcce98f279a8ebc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7cb00dde55d424d9dc7ee1fc139c6dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36275 } } peers { permanent_uuid: "a0a07d8cbfd141e49fcce98f279a8ebc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36255 } } peers { permanent_uuid: "126db552d7e645d597ecbab7867c8368" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43567 } }
I20260620 04:27:37.955091 15318 raft_consensus.cc:729] T fd554729413d4b398978509758144e49 P a0a07d8cbfd141e49fcce98f279a8ebc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0a07d8cbfd141e49fcce98f279a8ebc, State: Initialized, Role: FOLLOWER
I20260620 04:27:37.955639 15318 consensus_queue.cc:260] T fd554729413d4b398978509758144e49 P a0a07d8cbfd141e49fcce98f279a8ebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7cb00dde55d424d9dc7ee1fc139c6dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36275 } } peers { permanent_uuid: "a0a07d8cbfd141e49fcce98f279a8ebc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36255 } } peers { permanent_uuid: "126db552d7e645d597ecbab7867c8368" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43567 } }
I20260620 04:27:37.956760 15318 ts_tablet_manager.cc:1428] T fd554729413d4b398978509758144e49 P a0a07d8cbfd141e49fcce98f279a8ebc: Time spent starting tablet: real 0.006s user 0.004s sys 0.004s
W20260620 04:27:37.984921 15428 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:27:37.930291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36915 (local address 127.0.98.1:47897)
0620 04:27:37.930790 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:37.930844 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 04:27:37.931695 (+ 851us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:27:37.983850 (+ 52155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36915: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":125,"threads_started":1}
W20260620 04:27:37.985517 15426 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36915 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:36915: Client connection negotiation failed: client connection to 127.0.98.62:36915: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 04:27:37.986346 15428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:37.986033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36915 (local address 127.0.98.1:47145)
0620 04:27:37.986180 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:37.986268 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 04:27:37.987149 15428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:37.986914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36915 (local address 127.0.98.1:52299)
0620 04:27:37.987024 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:37.987081 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260620 04:27:37.987370 15426 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 04:27:38.052963 15429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:27:38.053254 15429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:27:38.053299 15429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:27:38.057327 15429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:27:38.057387 15429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:27:38.057449 15429 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 04:27:38.062722 15429 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:43567
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=43001
--tserver_master_addrs=127.0.98.62:36915
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15429
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:27:38.063803 15429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:27:38.064858 15429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:27:38.074443 15437 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:27:38.074411 15436 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:27:38.074476 15439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:27:38.074766 15429 server_base.cc:1034] running on GCE node
I20260620 04:27:38.075214 15429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:27:38.075906 15429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:27:38.077103 15429 hybrid_clock.cc:648] HybridClock initialized: now 1781929658077084 us; error 31 us; skew 500 ppm
Jun 20 04:27:38 dist-test-slave-nk6d krb5kdc[14118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929658, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:27:38.079874 15429 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 04:27:38.080785 15429 webserver.cc:460] Webserver started at http://127.0.98.2:43001/ using document root <none> and password file <none>
I20260620 04:27:38.081319 15429 fs_manager.cc:359] Metadata directory not provided
I20260620 04:27:38.081377 15429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:27:38.085347 15429 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 04:27:38.087414 15446 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:27:38.088827 15429 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 04:27:38.088929 15429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-1/wal
uuid: "126db552d7e645d597ecbab7867c8368"
format_stamp: "Formatted at 2026-06-20 04:27:35 on dist-test-slave-nk6d"
server_key: "41c6a4829c42a7f128ddc2edb2c9fa7f"
server_key_iv: "ad041497cafe3adab320c0e4650a0859"
server_key_version: "encryptionkey@0"
I20260620 04:27:38.089470 15429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:27:38.100380 15429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:27:38.103724 15429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:27:38.103891 15429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:27:38.104563 15429 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:27:38.105401 15429 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:27:38.105454 15429 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:27:38.105499 15429 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:27:38.105525 15429 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:27:38.114349 15429 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43567
I20260620 04:27:38.114380 15559 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43567 every 8 connection(s)
I20260620 04:27:38.115399 15429 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-1/data/info.pb
I20260620 04:27:38.119376 15562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:38.117779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36915 (local address 127.0.98.2:41229)
0620 04:27:38.118499 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:38.118614 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":137,"threads_started":1}
W20260620 04:27:38.119861 15560 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36915 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:36915: Client connection negotiation failed: client connection to 127.0.98.62:36915: connect: Connection refused (error 111)
I20260620 04:27:38.120527 15562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:38.120322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36915 (local address 127.0.98.2:46333)
0620 04:27:38.120423 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:38.120472 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260620 04:27:38.121174 15562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:38.120936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36915 (local address 127.0.98.2:59287)
0620 04:27:38.121050 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:38.121099 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 04:27:38.121348 15560 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 04:27:38.124573 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 15429
I20260620 04:27:38.124940 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:36275
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:36915
--webserver_port=42903
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:27:38.233728 15563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:27:38.233995 15563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:27:38.234038 15563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:27:38.238070 15563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:27:38.238137 15563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:27:38.238195 15563 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 04:27:38.243461 15563 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:36275
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=42903
--tserver_master_addrs=127.0.98.62:36915
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15563
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:27:38.244550 15563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:27:38.245541 15563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:27:38.254989 15568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:27:38.255055 15571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:27:38.255229 15563 server_base.cc:1034] running on GCE node
W20260620 04:27:38.255101 15569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:27:38.255690 15563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:27:38.256372 15563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:27:38.257534 15563 hybrid_clock.cc:648] HybridClock initialized: now 1781929658257501 us; error 37 us; skew 500 ppm
Jun 20 04:27:38 dist-test-slave-nk6d krb5kdc[14118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929658, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:27:38.260104 15563 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 04:27:38.260905 15563 webserver.cc:460] Webserver started at http://127.0.98.3:42903/ using document root <none> and password file <none>
I20260620 04:27:38.261401 15563 fs_manager.cc:359] Metadata directory not provided
I20260620 04:27:38.261456 15563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:27:38.265100 15563 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:27:38.266985 15578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:27:38.268217 15563 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:27:38.268321 15563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-2/wal
uuid: "a7cb00dde55d424d9dc7ee1fc139c6dc"
format_stamp: "Formatted at 2026-06-20 04:27:36 on dist-test-slave-nk6d"
server_key: "dedf2077f899b792f7a0bb8539d31e1e"
server_key_iv: "f2b819fcd140ae7eed49935e59d790d3"
server_key_version: "encryptionkey@0"
I20260620 04:27:38.268831 15563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:27:38.294814 15563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:27:38.297708 15563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:27:38.297880 15563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:27:38.298475 15563 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:27:38.299275 15563 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:27:38.299321 15563 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:27:38.299373 15563 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:27:38.299398 15563 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:27:38.308102 15563 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36275
I20260620 04:27:38.308118 15691 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36275 every 8 connection(s)
I20260620 04:27:38.309150 15563 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929496330398-392-0/minicluster-data/ts-2/data/info.pb
I20260620 04:27:38.311247 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 15563
I20260620 04:27:38.312950 15694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:38.311340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36915 (local address 127.0.98.3:57055)
0620 04:27:38.312070 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:38.312210 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":130,"threads_started":1}
W20260620 04:27:38.313359 15692 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36915 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:36915: Client connection negotiation failed: client connection to 127.0.98.62:36915: connect: Connection refused (error 111)
I20260620 04:27:38.313952 15694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:38.313739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36915 (local address 127.0.98.3:60055)
0620 04:27:38.313853 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:38.313898 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260620 04:27:38.314505 15694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:38.314320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36915 (local address 127.0.98.3:49539)
0620 04:27:38.314414 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:38.314456 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260620 04:27:38.314659 15692 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 04:27:38.988480 15696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:38.988038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36915 (local address 127.0.98.1:56899)
0620 04:27:38.988303 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:38.988366 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":78,"threads_started":1}
I20260620 04:27:39.122402 15697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:39.121966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36915 (local address 127.0.98.2:49261)
0620 04:27:39.122240 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:39.122300 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":88,"threads_started":1}
I20260620 04:27:39.315644 15698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:39.315246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36915 (local address 127.0.98.3:40273)
0620 04:27:39.315471 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:39.315541 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":69,"threads_started":1}
I20260620 04:27:39.744473 15699 raft_consensus.cc:482] T fd554729413d4b398978509758144e49 P a0a07d8cbfd141e49fcce98f279a8ebc [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:27:39.744680 15699 raft_consensus.cc:504] T fd554729413d4b398978509758144e49 P a0a07d8cbfd141e49fcce98f279a8ebc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7cb00dde55d424d9dc7ee1fc139c6dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36275 } } peers { permanent_uuid: "a0a07d8cbfd141e49fcce98f279a8ebc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36255 } } peers { permanent_uuid: "126db552d7e645d597ecbab7867c8368" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43567 } }
I20260620 04:27:39.745855 15699 leader_election.cc:288] T fd554729413d4b398978509758144e49 P a0a07d8cbfd141e49fcce98f279a8ebc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a7cb00dde55d424d9dc7ee1fc139c6dc (127.0.98.3:36275), 126db552d7e645d597ecbab7867c8368 (127.0.98.2:43567)
Jun 20 04:27:39 dist-test-slave-nk6d krb5kdc[14118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929657, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Jun 20 04:27:39 dist-test-slave-nk6d krb5kdc[14118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929657, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260620 04:27:39.758996 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:39.746242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42443 (local address 127.0.98.3:36275)
0620 04:27:39.746536 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0620 04:27:39.746561 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0620 04:27:39.746605 (+ 44us) server_negotiation.cc:407] Connection header received
0620 04:27:39.747461 (+ 856us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:27:39.747469 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:27:39.747866 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:27:39.748173 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:27:39.749408 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:39.750388 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:39.751584 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:39.752056 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:39.752200 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:27:39.754697 (+ 2497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:27:39.754702 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:27:39.754714 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:27:39.754745 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:27:39.756440 (+ 1695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:27:39.756962 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:27:39.756968 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:27:39.756973 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:27:39.757078 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:27:39.757447 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:27:39.757450 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:27:39.757451 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:27:39.757872 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:27:39.757988 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 04:27:39.758619 (+ 631us) server_negotiation.cc:299] Negotiation successful
0620 04:27:39.758857 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":83,"threads_started":1}
I20260620 04:27:39.759056 15701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:39.746134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43567 (local address 127.0.98.1:48155)
0620 04:27:39.746671 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:39.746684 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:27:39.746797 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:27:39.748248 (+ 1451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:27:39.748254 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:27:39.748782 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:27:39.749250 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:27:39.749267 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:27:39.750548 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:27:39.750550 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:27:39.751447 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:27:39.751456 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:27:39.752164 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:27:39.752167 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:27:39.752390 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:27:39.753021 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:27:39.753033 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:27:39.754827 (+ 1794us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:27:39.756558 (+ 1731us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:27:39.756561 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:27:39.756569 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:27:39.756862 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:27:39.757161 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:27:39.757165 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:27:39.757167 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:27:39.757339 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:27:39.758052 (+ 713us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:27:39.758055 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:27:39.758310 (+ 255us) client_negotiation.cc:769] Sending connection context
0620 04:27:39.758515 (+ 205us) client_negotiation.cc:240] Negotiation successful
0620 04:27:39.758906 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"mutex_wait_us":257,"spinlock_wait_cycles":15360,"thread_start_us":94,"threads_started":1}
I20260620 04:27:39.759057 15700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:39.746080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36275 (local address 127.0.98.1:42443)
0620 04:27:39.746328 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:39.746343 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 04:27:39.746447 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:27:39.748249 (+ 1802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:27:39.748255 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:27:39.748782 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:27:39.749251 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:27:39.749266 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:27:39.750540 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:27:39.750544 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:27:39.751445 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:27:39.751454 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:27:39.752162 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:27:39.752166 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:27:39.752393 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:27:39.752978 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:27:39.752993 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:27:39.754491 (+ 1498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:27:39.756547 (+ 2056us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:27:39.756552 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:27:39.756566 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:27:39.756857 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:27:39.757171 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:27:39.757175 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:27:39.757176 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:27:39.757348 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:27:39.757980 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:27:39.757986 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:27:39.758310 (+ 324us) client_negotiation.cc:769] Sending connection context
0620 04:27:39.758515 (+ 205us) client_negotiation.cc:240] Negotiation successful
0620 04:27:39.758843 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":158,"spinlock_wait_cycles":29568,"thread_start_us":62,"threads_started":1}
I20260620 04:27:39.759742 15703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:39.746347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48155 (local address 127.0.98.2:43567)
0620 04:27:39.746670 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0620 04:27:39.746684 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 04:27:39.746782 (+ 98us) server_negotiation.cc:407] Connection header received
0620 04:27:39.747556 (+ 774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:27:39.747564 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:27:39.747865 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:27:39.748174 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:27:39.749396 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:39.750445 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:39.751555 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:39.752057 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:39.752209 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:27:39.754986 (+ 2777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:27:39.754991 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:27:39.755002 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:27:39.755028 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:27:39.756461 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:27:39.756960 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:27:39.756965 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:27:39.756969 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:27:39.757071 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:27:39.757423 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:27:39.757426 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:27:39.757428 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:27:39.757898 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:27:39.758003 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 04:27:39.759313 (+ 1310us) server_negotiation.cc:299] Negotiation successful
0620 04:27:39.759569 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":49,"threads_started":1}
W20260620 04:27:39.761368 15313 leader_election.cc:341] T fd554729413d4b398978509758144e49 P a0a07d8cbfd141e49fcce98f279a8ebc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a7cb00dde55d424d9dc7ee1fc139c6dc (127.0.98.3:36275): Not found: Tablet not found: fd554729413d4b398978509758144e49
W20260620 04:27:39.761499 15314 leader_election.cc:341] T fd554729413d4b398978509758144e49 P a0a07d8cbfd141e49fcce98f279a8ebc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 126db552d7e645d597ecbab7867c8368 (127.0.98.2:43567): Not found: Tablet not found: fd554729413d4b398978509758144e49
I20260620 04:27:39.761595 15314 leader_election.cc:302] T fd554729413d4b398978509758144e49 P a0a07d8cbfd141e49fcce98f279a8ebc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0a07d8cbfd141e49fcce98f279a8ebc; no voters: 126db552d7e645d597ecbab7867c8368, a7cb00dde55d424d9dc7ee1fc139c6dc
I20260620 04:27:39.761855 15699 raft_consensus.cc:2726] T fd554729413d4b398978509758144e49 P a0a07d8cbfd141e49fcce98f279a8ebc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 04:27:39.989581 15700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:39.989228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36915 (local address 127.0.98.1:47303)
0620 04:27:39.989397 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:39.989471 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260620 04:27:40.123628 15704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:40.123135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36915 (local address 127.0.98.2:51911)
0620 04:27:40.123442 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:40.123510 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":86,"threads_started":1}
I20260620 04:27:40.316900 15705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:40.316489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36915 (local address 127.0.98.3:53911)
0620 04:27:40.316744 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:40.316804 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":86,"threads_started":1}
I20260620 04:27:40.990770 15706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:40.990351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36915 (local address 127.0.98.1:35527)
0620 04:27:40.990612 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:40.990674 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":74,"threads_started":1}
I20260620 04:27:41.124843 15707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:41.124427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36915 (local address 127.0.98.2:58899)
0620 04:27:41.124684 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:41.124747 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":87,"threads_started":1}
I20260620 04:27:41.318024 15708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:41.317594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36915 (local address 127.0.98.3:35819)
0620 04:27:41.317849 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:41.317914 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":70,"threads_started":1}
I20260620 04:27:41.992012 15709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:41.991533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36915 (local address 127.0.98.1:48753)
0620 04:27:41.991808 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:41.991872 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":79,"threads_started":1}
I20260620 04:27:42.002975 15710 raft_consensus.cc:482] T fd554729413d4b398978509758144e49 P a0a07d8cbfd141e49fcce98f279a8ebc [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:27:42.003111 15710 raft_consensus.cc:504] T fd554729413d4b398978509758144e49 P a0a07d8cbfd141e49fcce98f279a8ebc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7cb00dde55d424d9dc7ee1fc139c6dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36275 } } peers { permanent_uuid: "a0a07d8cbfd141e49fcce98f279a8ebc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36255 } } peers { permanent_uuid: "126db552d7e645d597ecbab7867c8368" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43567 } }
I20260620 04:27:42.003573 15710 leader_election.cc:288] T fd554729413d4b398978509758144e49 P a0a07d8cbfd141e49fcce98f279a8ebc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a7cb00dde55d424d9dc7ee1fc139c6dc (127.0.98.3:36275), 126db552d7e645d597ecbab7867c8368 (127.0.98.2:43567)
W20260620 04:27:42.004314 15314 leader_election.cc:341] T fd554729413d4b398978509758144e49 P a0a07d8cbfd141e49fcce98f279a8ebc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 126db552d7e645d597ecbab7867c8368 (127.0.98.2:43567): Not found: Tablet not found: fd554729413d4b398978509758144e49
W20260620 04:27:42.004428 15313 leader_election.cc:341] T fd554729413d4b398978509758144e49 P a0a07d8cbfd141e49fcce98f279a8ebc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a7cb00dde55d424d9dc7ee1fc139c6dc (127.0.98.3:36275): Not found: Tablet not found: fd554729413d4b398978509758144e49
I20260620 04:27:42.004557 15313 leader_election.cc:302] T fd554729413d4b398978509758144e49 P a0a07d8cbfd141e49fcce98f279a8ebc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0a07d8cbfd141e49fcce98f279a8ebc; no voters: 126db552d7e645d597ecbab7867c8368, a7cb00dde55d424d9dc7ee1fc139c6dc
I20260620 04:27:42.004693 15710 raft_consensus.cc:2726] T fd554729413d4b398978509758144e49 P a0a07d8cbfd141e49fcce98f279a8ebc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 04:27:42.125977 15711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:42.125566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36915 (local address 127.0.98.2:49065)
0620 04:27:42.125808 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:42.125864 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":76,"threads_started":1}
I20260620 04:27:42.319263 15712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:42.318803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36915 (local address 127.0.98.3:59905)
0620 04:27:42.319096 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:42.319165 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":76,"threads_started":1}
I20260620 04:27:42.993425 15713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:42.992960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36915 (local address 127.0.98.1:55079)
0620 04:27:42.993272 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:42.993331 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":80,"threads_started":1}
I20260620 04:27:43.127264 15714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:43.126817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36915 (local address 127.0.98.2:57365)
0620 04:27:43.127104 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:43.127168 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":90,"threads_started":1}
I20260620 04:27:43.320840 15715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:43.320340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36915 (local address 127.0.98.3:43659)
0620 04:27:43.320654 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:43.320726 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":84,"threads_started":1}
I20260620 04:27:43.994656 15716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:43.994223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36915 (local address 127.0.98.1:55349)
0620 04:27:43.994493 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:43.994560 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":74,"threads_started":1}
I20260620 04:27:44.128545 15717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:44.128078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36915 (local address 127.0.98.2:54413)
0620 04:27:44.128345 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:44.128407 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":74,"threads_started":1}
I20260620 04:27:44.322031 15718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:44.321581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36915 (local address 127.0.98.3:43125)
0620 04:27:44.321861 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:44.321924 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":75,"threads_started":1}
I20260620 04:27:44.586870 15719 raft_consensus.cc:482] T fd554729413d4b398978509758144e49 P a0a07d8cbfd141e49fcce98f279a8ebc [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:27:44.587009 15719 raft_consensus.cc:504] T fd554729413d4b398978509758144e49 P a0a07d8cbfd141e49fcce98f279a8ebc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7cb00dde55d424d9dc7ee1fc139c6dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36275 } } peers { permanent_uuid: "a0a07d8cbfd141e49fcce98f279a8ebc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36255 } } peers { permanent_uuid: "126db552d7e645d597ecbab7867c8368" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43567 } }
I20260620 04:27:44.587456 15719 leader_election.cc:288] T fd554729413d4b398978509758144e49 P a0a07d8cbfd141e49fcce98f279a8ebc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a7cb00dde55d424d9dc7ee1fc139c6dc (127.0.98.3:36275), 126db552d7e645d597ecbab7867c8368 (127.0.98.2:43567)
W20260620 04:27:44.588188 15314 leader_election.cc:341] T fd554729413d4b398978509758144e49 P a0a07d8cbfd141e49fcce98f279a8ebc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 126db552d7e645d597ecbab7867c8368 (127.0.98.2:43567): Not found: Tablet not found: fd554729413d4b398978509758144e49
W20260620 04:27:44.588322 15313 leader_election.cc:341] T fd554729413d4b398978509758144e49 P a0a07d8cbfd141e49fcce98f279a8ebc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a7cb00dde55d424d9dc7ee1fc139c6dc (127.0.98.3:36275): Not found: Tablet not found: fd554729413d4b398978509758144e49
I20260620 04:27:44.588395 15313 leader_election.cc:302] T fd554729413d4b398978509758144e49 P a0a07d8cbfd141e49fcce98f279a8ebc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0a07d8cbfd141e49fcce98f279a8ebc; no voters: 126db552d7e645d597ecbab7867c8368, a7cb00dde55d424d9dc7ee1fc139c6dc
I20260620 04:27:44.588546 15719 raft_consensus.cc:2726] T fd554729413d4b398978509758144e49 P a0a07d8cbfd141e49fcce98f279a8ebc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 04:27:44.995904 15720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:44.995404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36915 (local address 127.0.98.1:59869)
0620 04:27:44.995697 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:44.995762 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":91,"threads_started":1}
I20260620 04:27:45.129926 15721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:45.129458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36915 (local address 127.0.98.2:59821)
0620 04:27:45.129765 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:45.129827 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":77,"threads_started":1}
I20260620 04:27:45.323382 15722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:45.322856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36915 (local address 127.0.98.3:52441)
0620 04:27:45.323172 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:45.323266 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":73,"threads_started":1}
I20260620 04:27:45.997186 15723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:45.996727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36915 (local address 127.0.98.1:59547)
0620 04:27:45.997020 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:45.997089 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":83,"threads_started":1}
I20260620 04:27:46.131222 15724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:46.130747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36915 (local address 127.0.98.2:37549)
0620 04:27:46.131039 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:46.131116 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":80,"threads_started":1}
I20260620 04:27:46.324640 15725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:46.324178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36915 (local address 127.0.98.3:37057)
0620 04:27:46.324477 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:46.324542 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":73,"threads_started":1}
I20260620 04:27:46.387420 15726 raft_consensus.cc:482] T fd554729413d4b398978509758144e49 P a0a07d8cbfd141e49fcce98f279a8ebc [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:27:46.387559 15726 raft_consensus.cc:504] T fd554729413d4b398978509758144e49 P a0a07d8cbfd141e49fcce98f279a8ebc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7cb00dde55d424d9dc7ee1fc139c6dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36275 } } peers { permanent_uuid: "a0a07d8cbfd141e49fcce98f279a8ebc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36255 } } peers { permanent_uuid: "126db552d7e645d597ecbab7867c8368" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43567 } }
I20260620 04:27:46.387977 15726 leader_election.cc:288] T fd554729413d4b398978509758144e49 P a0a07d8cbfd141e49fcce98f279a8ebc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a7cb00dde55d424d9dc7ee1fc139c6dc (127.0.98.3:36275), 126db552d7e645d597ecbab7867c8368 (127.0.98.2:43567)
W20260620 04:27:46.388724 15313 leader_election.cc:341] T fd554729413d4b398978509758144e49 P a0a07d8cbfd141e49fcce98f279a8ebc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a7cb00dde55d424d9dc7ee1fc139c6dc (127.0.98.3:36275): Not found: Tablet not found: fd554729413d4b398978509758144e49
W20260620 04:27:46.388830 15314 leader_election.cc:341] T fd554729413d4b398978509758144e49 P a0a07d8cbfd141e49fcce98f279a8ebc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 126db552d7e645d597ecbab7867c8368 (127.0.98.2:43567): Not found: Tablet not found: fd554729413d4b398978509758144e49
I20260620 04:27:46.388914 15314 leader_election.cc:302] T fd554729413d4b398978509758144e49 P a0a07d8cbfd141e49fcce98f279a8ebc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0a07d8cbfd141e49fcce98f279a8ebc; no voters: 126db552d7e645d597ecbab7867c8368, a7cb00dde55d424d9dc7ee1fc139c6dc
I20260620 04:27:46.389043 15726 raft_consensus.cc:2726] T fd554729413d4b398978509758144e49 P a0a07d8cbfd141e49fcce98f279a8ebc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 04:27:46.998363 15727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:46.997921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36915 (local address 127.0.98.1:37397)
0620 04:27:46.998216 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:46.998270 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":88,"threads_started":1}
I20260620 04:27:47.132474 15728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:47.132040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36915 (local address 127.0.98.2:59917)
0620 04:27:47.132319 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:47.132380 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":72,"threads_started":1}
I20260620 04:27:47.325832 15729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:47.325404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36915 (local address 127.0.98.3:46555)
0620 04:27:47.325669 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:47.325732 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":88,"threads_started":1}
I20260620 04:27:47.999619 15730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:47.999182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36915 (local address 127.0.98.1:57017)
0620 04:27:47.999455 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:47.999516 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":90,"threads_started":1}
I20260620 04:27:48.133781 15731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:48.133286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36915 (local address 127.0.98.2:45291)
0620 04:27:48.133596 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:48.133664 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":96,"threads_started":1}
I20260620 04:27:48.326970 15732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:48.326591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36915 (local address 127.0.98.3:57283)
0620 04:27:48.326815 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:48.326877 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":72,"threads_started":1}
I20260620 04:27:48.684051 15733 raft_consensus.cc:482] T fd554729413d4b398978509758144e49 P a0a07d8cbfd141e49fcce98f279a8ebc [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:27:48.684181 15733 raft_consensus.cc:504] T fd554729413d4b398978509758144e49 P a0a07d8cbfd141e49fcce98f279a8ebc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7cb00dde55d424d9dc7ee1fc139c6dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36275 } } peers { permanent_uuid: "a0a07d8cbfd141e49fcce98f279a8ebc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36255 } } peers { permanent_uuid: "126db552d7e645d597ecbab7867c8368" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43567 } }
I20260620 04:27:48.684584 15733 leader_election.cc:288] T fd554729413d4b398978509758144e49 P a0a07d8cbfd141e49fcce98f279a8ebc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a7cb00dde55d424d9dc7ee1fc139c6dc (127.0.98.3:36275), 126db552d7e645d597ecbab7867c8368 (127.0.98.2:43567)
W20260620 04:27:48.685381 15313 leader_election.cc:341] T fd554729413d4b398978509758144e49 P a0a07d8cbfd141e49fcce98f279a8ebc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a7cb00dde55d424d9dc7ee1fc139c6dc (127.0.98.3:36275): Not found: Tablet not found: fd554729413d4b398978509758144e49
W20260620 04:27:48.685505 15314 leader_election.cc:341] T fd554729413d4b398978509758144e49 P a0a07d8cbfd141e49fcce98f279a8ebc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 126db552d7e645d597ecbab7867c8368 (127.0.98.2:43567): Not found: Tablet not found: fd554729413d4b398978509758144e49
I20260620 04:27:48.685566 15314 leader_election.cc:302] T fd554729413d4b398978509758144e49 P a0a07d8cbfd141e49fcce98f279a8ebc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0a07d8cbfd141e49fcce98f279a8ebc; no voters: 126db552d7e645d597ecbab7867c8368, a7cb00dde55d424d9dc7ee1fc139c6dc
I20260620 04:27:48.685689 15733 raft_consensus.cc:2726] T fd554729413d4b398978509758144e49 P a0a07d8cbfd141e49fcce98f279a8ebc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 04:27:49.000886 15734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:49.000383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36915 (local address 127.0.98.1:41411)
0620 04:27:49.000711 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:49.000778 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":103,"threads_started":1}
I20260620 04:27:49.135003 15735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:49.134564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36915 (local address 127.0.98.2:60689)
0620 04:27:49.134832 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:49.134911 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":79,"threads_started":1}
I20260620 04:27:49.328258 15736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:49.327786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36915 (local address 127.0.98.3:54611)
0620 04:27:49.328094 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:49.328160 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":93,"threads_started":1}
I20260620 04:27:50.002019 15737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:50.001611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36915 (local address 127.0.98.1:43885)
0620 04:27:50.001867 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:50.001923 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":75,"threads_started":1}
I20260620 04:27:50.136231 15738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:50.135764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36915 (local address 127.0.98.2:34755)
0620 04:27:50.136055 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:50.136116 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":72,"threads_started":1}
I20260620 04:27:50.329348 15739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:50.328928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36915 (local address 127.0.98.3:51221)
0620 04:27:50.329199 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:50.329256 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":74,"threads_started":1}
I20260620 04:27:51.003242 15740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:51.002803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36915 (local address 127.0.98.1:53815)
0620 04:27:51.003083 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:51.003145 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":74,"threads_started":1}
I20260620 04:27:51.137403 15741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:51.136980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36915 (local address 127.0.98.2:56353)
0620 04:27:51.137243 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:51.137305 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":81,"threads_started":1}
I20260620 04:27:51.330578 15742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:51.330134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36915 (local address 127.0.98.3:37753)
0620 04:27:51.330412 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:51.330480 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":81,"threads_started":1}
I20260620 04:27:52.004530 15743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:52.004059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36915 (local address 127.0.98.1:46911)
0620 04:27:52.004348 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:52.004427 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":71,"threads_started":1}
I20260620 04:27:52.138581 15744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:52.138153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36915 (local address 127.0.98.2:55553)
0620 04:27:52.138430 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:52.138486 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":91,"threads_started":1}
I20260620 04:27:52.317656 15745 raft_consensus.cc:482] T fd554729413d4b398978509758144e49 P a0a07d8cbfd141e49fcce98f279a8ebc [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:27:52.317786 15745 raft_consensus.cc:504] T fd554729413d4b398978509758144e49 P a0a07d8cbfd141e49fcce98f279a8ebc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7cb00dde55d424d9dc7ee1fc139c6dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36275 } } peers { permanent_uuid: "a0a07d8cbfd141e49fcce98f279a8ebc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36255 } } peers { permanent_uuid: "126db552d7e645d597ecbab7867c8368" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43567 } }
I20260620 04:27:52.318177 15745 leader_election.cc:288] T fd554729413d4b398978509758144e49 P a0a07d8cbfd141e49fcce98f279a8ebc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a7cb00dde55d424d9dc7ee1fc139c6dc (127.0.98.3:36275), 126db552d7e645d597ecbab7867c8368 (127.0.98.2:43567)
W20260620 04:27:52.318876 15313 leader_election.cc:341] T fd554729413d4b398978509758144e49 P a0a07d8cbfd141e49fcce98f279a8ebc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a7cb00dde55d424d9dc7ee1fc139c6dc (127.0.98.3:36275): Not found: Tablet not found: fd554729413d4b398978509758144e49
W20260620 04:27:52.318958 15314 leader_election.cc:341] T fd554729413d4b398978509758144e49 P a0a07d8cbfd141e49fcce98f279a8ebc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 126db552d7e645d597ecbab7867c8368 (127.0.98.2:43567): Not found: Tablet not found: fd554729413d4b398978509758144e49
I20260620 04:27:52.319063 15314 leader_election.cc:302] T fd554729413d4b398978509758144e49 P a0a07d8cbfd141e49fcce98f279a8ebc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0a07d8cbfd141e49fcce98f279a8ebc; no voters: 126db552d7e645d597ecbab7867c8368, a7cb00dde55d424d9dc7ee1fc139c6dc
I20260620 04:27:52.319229 15745 raft_consensus.cc:2726] T fd554729413d4b398978509758144e49 P a0a07d8cbfd141e49fcce98f279a8ebc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 04:27:52.331748 15746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:52.331233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36915 (local address 127.0.98.3:37785)
0620 04:27:52.331534 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:52.331617 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":72,"threads_started":1}
I20260620 04:27:53.006017 15747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:53.005538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36915 (local address 127.0.98.1:53855)
0620 04:27:53.005857 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:53.005916 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":91,"threads_started":1}
I20260620 04:27:53.140064 15748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:53.139533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36915 (local address 127.0.98.2:57567)
0620 04:27:53.139861 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:53.139924 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":86,"threads_started":1}
I20260620 04:27:53.318048 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 15262
I20260620 04:27:53.323065 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 15429
I20260620 04:27:53.327497 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 15563
2026-06-20T04:27:53Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19098 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:27:53 dist-test-slave-nk6d krb5kdc[15751](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:27:53 dist-test-slave-nk6d krb5kdc[15751](info): set up 2 sockets
Jun 20 04:27:53 dist-test-slave-nk6d krb5kdc[15751](info): commencing operation
krb5kdc: starting...
W20260620 04:27:54.279909 392 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/krb5kdc/test-user.keytab.
Jun 20 04:27:54 dist-test-slave-nk6d krb5kdc[15751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929674, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:27:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:27:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 04:27:54.390162 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39499
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39499
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:27:54.496749 15764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:27:54.497002 15764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:27:54.497058 15764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:27:54.501005 15764 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 04:27:54.501058 15764 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 04:27:54.501080 15764 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 04:27:54.501096 15764 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:27:54.501121 15764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:27:54.501153 15764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:27:54.501184 15764 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:27:54.501209 15764 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:27:54.506423 15764 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39499
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39499
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15764
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:27:54.507504 15764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:27:54.508572 15764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 04:27:54.517156 15764 server_base.cc:1034] running on GCE node
W20260620 04:27:54.517179 15769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:27:54.517179 15770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:27:54.517202 15772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:27:54.518019 15764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:27:54.519172 15764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:27:54.520351 15764 hybrid_clock.cc:648] HybridClock initialized: now 1781929674520327 us; error 36 us; skew 500 ppm
Jun 20 04:27:54 dist-test-slave-nk6d krb5kdc[15751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929674, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:27:54.523260 15764 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 04:27:54.524255 15764 webserver.cc:460] Webserver started at http://127.0.98.62:38563/ using document root <none> and password file <none>
I20260620 04:27:54.524811 15764 fs_manager.cc:359] Metadata directory not provided
I20260620 04:27:54.524878 15764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:27:54.525080 15764 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:27:54.527110 15764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/master-0/data/instance:
uuid: "cccb8275856c44bbb18eacd620970116"
format_stamp: "Formatted at 2026-06-20 04:27:54 on dist-test-slave-nk6d"
server_key: "130fb43b0ecebb08308b5116486efb34"
server_key_iv: "fcf121ace544218ffbc10f663267120a"
server_key_version: "encryptionkey@0"
I20260620 04:27:54.527643 15764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/master-0/wal/instance:
uuid: "cccb8275856c44bbb18eacd620970116"
format_stamp: "Formatted at 2026-06-20 04:27:54 on dist-test-slave-nk6d"
server_key: "130fb43b0ecebb08308b5116486efb34"
server_key_iv: "fcf121ace544218ffbc10f663267120a"
server_key_version: "encryptionkey@0"
I20260620 04:27:54.531343 15764 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 04:27:54.533851 15779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:27:54.535180 15764 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 04:27:54.535296 15764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/master-0/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/master-0/wal
uuid: "cccb8275856c44bbb18eacd620970116"
format_stamp: "Formatted at 2026-06-20 04:27:54 on dist-test-slave-nk6d"
server_key: "130fb43b0ecebb08308b5116486efb34"
server_key_iv: "fcf121ace544218ffbc10f663267120a"
server_key_version: "encryptionkey@0"
I20260620 04:27:54.535399 15764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:27:54.560210 15764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:27:54.563211 15764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:27:54.563391 15764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:27:54.570039 15764 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39499
I20260620 04:27:54.570375 15831 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39499 every 8 connection(s)
I20260620 04:27:54.571050 15764 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:27:54.573733 15832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:27:54.575991 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 15764
I20260620 04:27:54.576107 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/master-0/wal/instance
I20260620 04:27:54.576403 392 external_mini_cluster.cc:1428] Setting key 39259e1124e491221aa17b3c6244d11e
I20260620 04:27:54.579746 15832 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cccb8275856c44bbb18eacd620970116: Bootstrap starting.
I20260620 04:27:54.581872 15832 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cccb8275856c44bbb18eacd620970116: Neither blocks nor log segments found. Creating new log.
I20260620 04:27:54.582564 15832 log.cc:824] T 00000000000000000000000000000000 P cccb8275856c44bbb18eacd620970116: Log is configured to *not* fsync() on all Append() calls
I20260620 04:27:54.584360 15832 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cccb8275856c44bbb18eacd620970116: No bootstrap required, opened a new log
Jun 20 04:27:54 dist-test-slave-nk6d krb5kdc[15751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929674, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:27:54.587242 15832 raft_consensus.cc:348] T 00000000000000000000000000000000 P cccb8275856c44bbb18eacd620970116 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cccb8275856c44bbb18eacd620970116" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39499 } }
I20260620 04:27:54.587451 15832 raft_consensus.cc:374] T 00000000000000000000000000000000 P cccb8275856c44bbb18eacd620970116 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:27:54.587507 15832 raft_consensus.cc:729] T 00000000000000000000000000000000 P cccb8275856c44bbb18eacd620970116 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cccb8275856c44bbb18eacd620970116, State: Initialized, Role: FOLLOWER
I20260620 04:27:54.587890 15832 consensus_queue.cc:260] T 00000000000000000000000000000000 P cccb8275856c44bbb18eacd620970116 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cccb8275856c44bbb18eacd620970116" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39499 } }
I20260620 04:27:54.588043 15832 raft_consensus.cc:388] T 00000000000000000000000000000000 P cccb8275856c44bbb18eacd620970116 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:27:54.588109 15832 raft_consensus.cc:482] T 00000000000000000000000000000000 P cccb8275856c44bbb18eacd620970116 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:27:54.588201 15832 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cccb8275856c44bbb18eacd620970116 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:27:54.589006 15832 raft_consensus.cc:504] T 00000000000000000000000000000000 P cccb8275856c44bbb18eacd620970116 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cccb8275856c44bbb18eacd620970116" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39499 } }
I20260620 04:27:54.589320 15832 leader_election.cc:302] T 00000000000000000000000000000000 P cccb8275856c44bbb18eacd620970116 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cccb8275856c44bbb18eacd620970116; no voters:
I20260620 04:27:54.589640 15832 leader_election.cc:288] T 00000000000000000000000000000000 P cccb8275856c44bbb18eacd620970116 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:27:54.589740 15837 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cccb8275856c44bbb18eacd620970116 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:27:54.589892 15837 raft_consensus.cc:686] T 00000000000000000000000000000000 P cccb8275856c44bbb18eacd620970116 [term 1 LEADER]: Becoming Leader. State: Replica: cccb8275856c44bbb18eacd620970116, State: Running, Role: LEADER
I20260620 04:27:54.590237 15837 consensus_queue.cc:237] T 00000000000000000000000000000000 P cccb8275856c44bbb18eacd620970116 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cccb8275856c44bbb18eacd620970116" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39499 } }
I20260620 04:27:54.590776 15835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:54.577845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56646 (local address 127.0.98.62:39499)
0620 04:27:54.578261 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0620 04:27:54.578283 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0620 04:27:54.578316 (+ 33us) server_negotiation.cc:407] Connection header received
0620 04:27:54.578956 (+ 640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:27:54.578973 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:27:54.579302 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:27:54.579604 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:27:54.580396 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:54.581352 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:54.582196 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:54.582658 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:54.582788 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:27:54.585415 (+ 2627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:27:54.585420 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:27:54.585433 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:27:54.585458 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:27:54.587521 (+ 2063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:27:54.588016 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:27:54.588029 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:27:54.588036 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:27:54.588173 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:27:54.588420 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:27:54.588423 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:27:54.588425 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:27:54.588846 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:27:54.589041 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0620 04:27:54.589591 (+ 550us) server_negotiation.cc:299] Negotiation successful
0620 04:27:54.589871 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":173,"threads_started":1}
I20260620 04:27:54.590873 15832 sys_catalog.cc:564] T 00000000000000000000000000000000 P cccb8275856c44bbb18eacd620970116 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:27:54.591893 15838 sys_catalog.cc:455] T 00000000000000000000000000000000 P cccb8275856c44bbb18eacd620970116 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cccb8275856c44bbb18eacd620970116" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cccb8275856c44bbb18eacd620970116" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39499 } } }
I20260620 04:27:54.592036 15838 sys_catalog.cc:458] T 00000000000000000000000000000000 P cccb8275856c44bbb18eacd620970116 [sys.catalog]: This master's current role is: LEADER
I20260620 04:27:54.592401 15839 sys_catalog.cc:455] T 00000000000000000000000000000000 P cccb8275856c44bbb18eacd620970116 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cccb8275856c44bbb18eacd620970116. Latest consensus state: current_term: 1 leader_uuid: "cccb8275856c44bbb18eacd620970116" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cccb8275856c44bbb18eacd620970116" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39499 } } }
I20260620 04:27:54.592542 15839 sys_catalog.cc:458] T 00000000000000000000000000000000 P cccb8275856c44bbb18eacd620970116 [sys.catalog]: This master's current role is: LEADER
I20260620 04:27:54.593114 15846 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:27:54.595711 15846 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:27:54.600472 15846 catalog_manager.cc:1347] Generated new cluster ID: 6722e5d138fc44cebacf726ef2c97bb5
I20260620 04:27:54.600529 15846 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:27:54.610633 15846 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:27:54.611802 15846 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:27:54.617322 15846 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cccb8275856c44bbb18eacd620970116: Generated new TSK 0
I20260620 04:27:54.618072 15846 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 04:27:54.648319 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39499
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:27:54.754664 15858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:27:54.754901 15858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:27:54.754959 15858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:27:54.758879 15858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:27:54.758934 15858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:27:54.759019 15858 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 04:27:54.764086 15858 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39499
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15858
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:27:54.765064 15858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:27:54.766028 15858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:27:54.775152 15864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:27:54.775158 15863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:27:54.775161 15866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:27:54.775278 15858 server_base.cc:1034] running on GCE node
I20260620 04:27:54.775789 15858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:27:54.776430 15858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:27:54.777570 15858 hybrid_clock.cc:648] HybridClock initialized: now 1781929674777551 us; error 38 us; skew 500 ppm
Jun 20 04:27:54 dist-test-slave-nk6d krb5kdc[15751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929674, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:27:54.780195 15858 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 04:27:54.780997 15858 webserver.cc:460] Webserver started at http://127.0.98.1:44573/ using document root <none> and password file <none>
I20260620 04:27:54.781504 15858 fs_manager.cc:359] Metadata directory not provided
I20260620 04:27:54.781569 15858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:27:54.781761 15858 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:27:54.783689 15858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-0/data/instance:
uuid: "54e13c6e85e747aea02cf40981ed4499"
format_stamp: "Formatted at 2026-06-20 04:27:54 on dist-test-slave-nk6d"
server_key: "cf70bf115049eb5d5d33ad49fef6187c"
server_key_iv: "b6361780b97fbbe09e78d86cb838f684"
server_key_version: "encryptionkey@0"
I20260620 04:27:54.784240 15858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-0/wal/instance:
uuid: "54e13c6e85e747aea02cf40981ed4499"
format_stamp: "Formatted at 2026-06-20 04:27:54 on dist-test-slave-nk6d"
server_key: "cf70bf115049eb5d5d33ad49fef6187c"
server_key_iv: "b6361780b97fbbe09e78d86cb838f684"
server_key_version: "encryptionkey@0"
I20260620 04:27:54.787719 15858 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260620 04:27:54.790026 15873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:27:54.791230 15858 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:27:54.791335 15858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-0/wal
uuid: "54e13c6e85e747aea02cf40981ed4499"
format_stamp: "Formatted at 2026-06-20 04:27:54 on dist-test-slave-nk6d"
server_key: "cf70bf115049eb5d5d33ad49fef6187c"
server_key_iv: "b6361780b97fbbe09e78d86cb838f684"
server_key_version: "encryptionkey@0"
I20260620 04:27:54.791428 15858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:27:54.808527 15858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:27:54.811262 15858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:27:54.811419 15858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:27:54.811951 15858 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:27:54.812736 15858 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:27:54.812780 15858 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:27:54.812815 15858 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:27:54.812842 15858 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:27:54.820783 15858 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34337
I20260620 04:27:54.820816 15986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34337 every 8 connection(s)
I20260620 04:27:54.821585 15858 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-0/data/info.pb
I20260620 04:27:54.824258 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 15858
I20260620 04:27:54.824368 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-0/wal/instance
I20260620 04:27:54.824683 392 external_mini_cluster.cc:1428] Setting key e55a953b7a63c17777198763d4dc3256
Jun 20 04:27:54 dist-test-slave-nk6d krb5kdc[15751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929674, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 04:27:54.835033 15835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:54.823191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35891 (local address 127.0.98.62:39499)
0620 04:27:54.823314 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 04:27:54.823317 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:27:54.824125 (+ 808us) server_negotiation.cc:407] Connection header received
0620 04:27:54.825603 (+ 1478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:27:54.825606 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:27:54.825649 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:27:54.825729 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:27:54.827178 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:54.827843 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:54.829201 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:54.829415 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:54.829503 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:27:54.831897 (+ 2394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:27:54.831901 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:27:54.831903 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:27:54.831925 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:27:54.833205 (+ 1280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:27:54.833647 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:27:54.833650 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:27:54.833652 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:27:54.833736 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:27:54.834031 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:27:54.834034 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:27:54.834036 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:27:54.834205 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:27:54.834286 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 04:27:54.834752 (+ 466us) server_negotiation.cc:299] Negotiation successful
0620 04:27:54.834872 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 04:27:54.835845 15989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:54.823536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39499 (local address 127.0.98.1:35891)
0620 04:27:54.823946 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:54.824023 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0620 04:27:54.825409 (+ 1386us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:27:54.825891 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:27:54.825900 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:27:54.826359 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:27:54.827032 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:27:54.827050 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:27:54.827984 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:27:54.827988 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:27:54.829110 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:27:54.829120 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:27:54.829555 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:27:54.829559 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:27:54.829721 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:27:54.830232 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:27:54.830248 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:27:54.831747 (+ 1499us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:27:54.833308 (+ 1561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:27:54.833314 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:27:54.833328 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:27:54.833559 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:27:54.833816 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:27:54.833819 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:27:54.833821 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:27:54.833947 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:27:54.834299 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:27:54.834305 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:27:54.834537 (+ 232us) client_negotiation.cc:769] Sending connection context
0620 04:27:54.834696 (+ 159us) client_negotiation.cc:240] Negotiation successful
0620 04:27:54.835001 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":94,"threads_started":1}
I20260620 04:27:54.836998 15987 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39499
I20260620 04:27:54.837248 15987 heartbeater.cc:464] Registering TS with master...
I20260620 04:27:54.837698 15987 heartbeater.cc:511] Master 127.0.98.62:39499 requested a full tablet report, sending...
I20260620 04:27:54.839208 15796 ts_manager.cc:194] Registered new tserver with Master: 54e13c6e85e747aea02cf40981ed4499 (127.0.98.1:34337)
I20260620 04:27:54.840360 15796 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:35891
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 04:27:54.848786 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39499
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:27:54.954943 15992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:27:54.955199 15992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:27:54.955257 15992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:27:54.959287 15992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:27:54.959347 15992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:27:54.959426 15992 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 04:27:54.964512 15992 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39499
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15992
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:27:54.965494 15992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:27:54.966447 15992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:27:54.975600 15997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:27:54.975620 15998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:27:54.975718 15992 server_base.cc:1034] running on GCE node
W20260620 04:27:54.975620 16000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:27:54.976218 15992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:27:54.976797 15992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:27:54.977993 15992 hybrid_clock.cc:648] HybridClock initialized: now 1781929674977958 us; error 44 us; skew 500 ppm
Jun 20 04:27:54 dist-test-slave-nk6d krb5kdc[15751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929674, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:27:54.980639 15992 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 04:27:54.981447 15992 webserver.cc:460] Webserver started at http://127.0.98.2:43673/ using document root <none> and password file <none>
I20260620 04:27:54.981963 15992 fs_manager.cc:359] Metadata directory not provided
I20260620 04:27:54.982025 15992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:27:54.982215 15992 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:27:54.984263 15992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-1/data/instance:
uuid: "a546aa48667e483bbbde10dfcdf180ff"
format_stamp: "Formatted at 2026-06-20 04:27:54 on dist-test-slave-nk6d"
server_key: "417962fc6e424d7083ac63c3da11ea99"
server_key_iv: "4c9b56092034d7f31b4d4353ba22ed2b"
server_key_version: "encryptionkey@0"
I20260620 04:27:54.984879 15992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a546aa48667e483bbbde10dfcdf180ff"
format_stamp: "Formatted at 2026-06-20 04:27:54 on dist-test-slave-nk6d"
server_key: "417962fc6e424d7083ac63c3da11ea99"
server_key_iv: "4c9b56092034d7f31b4d4353ba22ed2b"
server_key_version: "encryptionkey@0"
I20260620 04:27:54.988448 15992 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 04:27:54.990756 16007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:27:54.992033 15992 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:27:54.992139 15992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-1/wal
uuid: "a546aa48667e483bbbde10dfcdf180ff"
format_stamp: "Formatted at 2026-06-20 04:27:54 on dist-test-slave-nk6d"
server_key: "417962fc6e424d7083ac63c3da11ea99"
server_key_iv: "4c9b56092034d7f31b4d4353ba22ed2b"
server_key_version: "encryptionkey@0"
I20260620 04:27:54.992236 15992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:27:55.010035 15992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:27:55.013175 15992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:27:55.013363 15992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:27:55.013979 15992 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:27:55.014829 15992 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:27:55.014875 15992 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:27:55.014927 15992 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:27:55.014957 15992 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:27:55.023329 15992 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38671
I20260620 04:27:55.023352 16120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38671 every 8 connection(s)
I20260620 04:27:55.024238 15992 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-1/data/info.pb
I20260620 04:27:55.024442 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 15992
I20260620 04:27:55.024542 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-1/wal/instance
I20260620 04:27:55.024881 392 external_mini_cluster.cc:1428] Setting key 6b5348d64468675aa98649e9f03bc0b3
Jun 20 04:27:55 dist-test-slave-nk6d krb5kdc[15751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929674, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260620 04:27:55.038555 15835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:55.026049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45061 (local address 127.0.98.62:39499)
0620 04:27:55.026174 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 04:27:55.026178 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:27:55.027246 (+ 1068us) server_negotiation.cc:407] Connection header received
0620 04:27:55.028694 (+ 1448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:27:55.028697 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:27:55.028749 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:27:55.028819 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:27:55.029972 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:55.030743 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:55.031976 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:55.032331 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:55.032446 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:27:55.034821 (+ 2375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:27:55.034826 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:27:55.034829 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:27:55.034859 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:27:55.036288 (+ 1429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:27:55.036831 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:27:55.036834 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:27:55.036835 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:27:55.036919 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:27:55.037283 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:27:55.037287 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:27:55.037289 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:27:55.037536 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:27:55.037651 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 04:27:55.038305 (+ 654us) server_negotiation.cc:299] Negotiation successful
0620 04:27:55.038422 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 04:27:55.039587 16123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:55.026594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39499 (local address 127.0.98.2:45061)
0620 04:27:55.027069 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:55.027128 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 04:27:55.028488 (+ 1360us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:27:55.028986 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:27:55.028994 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:27:55.029339 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:27:55.029830 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:27:55.029843 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:27:55.030878 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:27:55.030881 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:27:55.031831 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:27:55.031840 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:27:55.032471 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:27:55.032474 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:27:55.032636 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:27:55.033171 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:27:55.033186 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:27:55.034659 (+ 1473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:27:55.036393 (+ 1734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:27:55.036398 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:27:55.036411 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:27:55.036716 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:27:55.037014 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:27:55.037016 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:27:55.037017 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:27:55.037166 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:27:55.037650 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:27:55.037656 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:27:55.037973 (+ 317us) client_negotiation.cc:769] Sending connection context
0620 04:27:55.038235 (+ 262us) client_negotiation.cc:240] Negotiation successful
0620 04:27:55.038607 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":97,"threads_started":1}
I20260620 04:27:55.040666 16121 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39499
I20260620 04:27:55.040920 16121 heartbeater.cc:464] Registering TS with master...
I20260620 04:27:55.041501 16121 heartbeater.cc:511] Master 127.0.98.62:39499 requested a full tablet report, sending...
I20260620 04:27:55.042681 15796 ts_manager.cc:194] Registered new tserver with Master: a546aa48667e483bbbde10dfcdf180ff (127.0.98.2:38671)
I20260620 04:27:55.043285 15796 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:45061
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 04:27:55.052508 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39499
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:27:55.160802 16126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:27:55.161072 16126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:27:55.161144 16126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:27:55.165171 16126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:27:55.165238 16126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:27:55.165319 16126 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 04:27:55.170571 16126 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39499
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16126
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:27:55.171607 16126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:27:55.172672 16126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:27:55.182010 16132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:27:55.182008 16134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:27:55.182236 16126 server_base.cc:1034] running on GCE node
W20260620 04:27:55.182010 16131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:27:55.182646 16126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:27:55.183297 16126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:27:55.184458 16126 hybrid_clock.cc:648] HybridClock initialized: now 1781929675184428 us; error 43 us; skew 500 ppm
Jun 20 04:27:55 dist-test-slave-nk6d krb5kdc[15751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929675, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:27:55.187136 16126 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 04:27:55.188051 16126 webserver.cc:460] Webserver started at http://127.0.98.3:44699/ using document root <none> and password file <none>
I20260620 04:27:55.188563 16126 fs_manager.cc:359] Metadata directory not provided
I20260620 04:27:55.188627 16126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:27:55.188810 16126 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:27:55.190758 16126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-2/data/instance:
uuid: "ce2d44319f6d48ee894f1d5bfc293a78"
format_stamp: "Formatted at 2026-06-20 04:27:55 on dist-test-slave-nk6d"
server_key: "325daa7963f8f69e62f6445288a6a072"
server_key_iv: "815942cb08a3453b062f0e0623e31ad6"
server_key_version: "encryptionkey@0"
I20260620 04:27:55.191263 16126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ce2d44319f6d48ee894f1d5bfc293a78"
format_stamp: "Formatted at 2026-06-20 04:27:55 on dist-test-slave-nk6d"
server_key: "325daa7963f8f69e62f6445288a6a072"
server_key_iv: "815942cb08a3453b062f0e0623e31ad6"
server_key_version: "encryptionkey@0"
I20260620 04:27:55.194880 16126 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 04:27:55.197357 16141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:27:55.198537 16126 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 04:27:55.198639 16126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-2/wal
uuid: "ce2d44319f6d48ee894f1d5bfc293a78"
format_stamp: "Formatted at 2026-06-20 04:27:55 on dist-test-slave-nk6d"
server_key: "325daa7963f8f69e62f6445288a6a072"
server_key_iv: "815942cb08a3453b062f0e0623e31ad6"
server_key_version: "encryptionkey@0"
I20260620 04:27:55.198755 16126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:27:55.220777 16126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:27:55.223613 16126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:27:55.223796 16126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:27:55.224385 16126 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:27:55.225186 16126 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:27:55.225232 16126 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:27:55.225283 16126 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:27:55.225312 16126 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:27:55.233721 16126 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39419
I20260620 04:27:55.233762 16254 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39419 every 8 connection(s)
I20260620 04:27:55.234601 16126 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-2/data/info.pb
I20260620 04:27:55.238701 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 16126
I20260620 04:27:55.238819 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-2/wal/instance
I20260620 04:27:55.239125 392 external_mini_cluster.cc:1428] Setting key 1877805349d2dcb448dc6e78a28c8a58
Jun 20 04:27:55 dist-test-slave-nk6d krb5kdc[15751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929675, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:27:55.247605 15835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:55.236414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37113 (local address 127.0.98.62:39499)
0620 04:27:55.236560 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 04:27:55.236565 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:27:55.237440 (+ 875us) server_negotiation.cc:407] Connection header received
0620 04:27:55.238714 (+ 1274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:27:55.238717 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:27:55.238766 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:27:55.238858 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:27:55.240248 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:55.240853 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:55.242107 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:55.242326 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:55.242414 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:27:55.244695 (+ 2281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:27:55.244698 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:27:55.244700 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:27:55.244724 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:27:55.245962 (+ 1238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:27:55.246408 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:27:55.246410 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:27:55.246411 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:27:55.246483 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:27:55.246730 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:27:55.246731 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:27:55.246732 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:27:55.246901 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:27:55.246968 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 04:27:55.247381 (+ 413us) server_negotiation.cc:299] Negotiation successful
0620 04:27:55.247472 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 04:27:55.248409 16257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:55.236821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39499 (local address 127.0.98.3:37113)
0620 04:27:55.237287 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:55.237331 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 04:27:55.238471 (+ 1140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:27:55.239048 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:27:55.239057 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:27:55.239508 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:27:55.240108 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:27:55.240128 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:27:55.240986 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:27:55.240990 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:27:55.241963 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:27:55.241973 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:27:55.242458 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:27:55.242461 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:27:55.242597 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:27:55.243157 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:27:55.243173 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:27:55.244543 (+ 1370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:27:55.246062 (+ 1519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:27:55.246067 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:27:55.246081 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:27:55.246324 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:27:55.246543 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:27:55.246545 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:27:55.246546 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:27:55.246668 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:27:55.246968 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:27:55.246973 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:27:55.247200 (+ 227us) client_negotiation.cc:769] Sending connection context
0620 04:27:55.247348 (+ 148us) client_negotiation.cc:240] Negotiation successful
0620 04:27:55.247578 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":108,"threads_started":1}
I20260620 04:27:55.249419 16255 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39499
I20260620 04:27:55.249644 16255 heartbeater.cc:464] Registering TS with master...
I20260620 04:27:55.250079 16255 heartbeater.cc:511] Master 127.0.98.62:39499 requested a full tablet report, sending...
I20260620 04:27:55.251001 15796 ts_manager.cc:194] Registered new tserver with Master: ce2d44319f6d48ee894f1d5bfc293a78 (127.0.98.3:39419)
I20260620 04:27:55.251560 15796 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:37113
I20260620 04:27:55.252851 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 04:27:55.261328 15835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:55.254759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56648 (local address 127.0.98.62:39499)
0620 04:27:55.254871 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 04:27:55.254875 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:27:55.255001 (+ 126us) server_negotiation.cc:407] Connection header received
0620 04:27:55.255111 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:27:55.255114 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:27:55.255155 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:27:55.255217 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:27:55.255848 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:55.256396 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:55.257187 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:55.257406 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:55.257488 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:27:55.258659 (+ 1171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:27:55.258663 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:27:55.258666 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:27:55.258689 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:27:55.259895 (+ 1206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:27:55.260272 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:27:55.260275 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:27:55.260277 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:27:55.260345 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:27:55.260571 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:27:55.260574 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:27:55.260575 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:27:55.260773 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:27:55.260886 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 04:27:55.261088 (+ 202us) server_negotiation.cc:299] Negotiation successful
0620 04:27:55.261189 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 04:27:55.263868 15795 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56648:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 04:27:55.266775 15795 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 04:27:55.281622 16267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:55.275033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38671 (local address 127.0.0.1:33322)
0620 04:27:55.276121 (+ 1088us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:55.276172 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 04:27:55.276326 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:27:55.277027 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:27:55.277030 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:27:55.277046 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:27:55.277119 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:27:55.277124 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:27:55.278556 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:27:55.278559 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:27:55.280225 (+ 1666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:27:55.280237 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:27:55.281261 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:27:55.281265 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:27:55.281345 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:27:55.281364 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 04:27:55.281411 (+ 47us) client_negotiation.cc:240] Negotiation successful
0620 04:27:55.281467 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":889,"mutex_wait_us":169,"thread_start_us":101,"threads_started":1}
I20260620 04:27:55.281683 16269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:55.275296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33322 (local address 127.0.98.2:38671)
0620 04:27:55.275850 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0620 04:27:55.275861 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 04:27:55.276465 (+ 604us) server_negotiation.cc:407] Connection header received
0620 04:27:55.276531 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:27:55.276535 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:27:55.276713 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:27:55.276856 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:27:55.277238 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:55.278412 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:55.280348 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:55.281153 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:55.281255 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:27:55.281392 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 04:27:55.281445 (+ 53us) server_negotiation.cc:299] Negotiation successful
0620 04:27:55.281535 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":113,"threads_started":1}
I20260620 04:27:55.281776 16271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:55.275855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43480 (local address 127.0.98.3:39419)
0620 04:27:55.276386 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0620 04:27:55.276399 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 04:27:55.276422 (+ 23us) server_negotiation.cc:407] Connection header received
0620 04:27:55.276496 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:27:55.276501 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:27:55.276708 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:27:55.276875 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:27:55.277186 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:55.278412 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:55.280377 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:55.280923 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:55.281021 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:27:55.281175 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 04:27:55.281542 (+ 367us) server_negotiation.cc:299] Negotiation successful
0620 04:27:55.281632 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":177,"threads_started":1}
I20260620 04:27:55.282006 16268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:55.275043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39419 (local address 127.0.0.1:43480)
0620 04:27:55.276305 (+ 1262us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:55.276319 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:27:55.276396 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:27:55.276889 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:27:55.276894 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:27:55.276914 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:27:55.277053 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:27:55.277060 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:27:55.278556 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:27:55.278559 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:27:55.280247 (+ 1688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:27:55.280253 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:27:55.281045 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:27:55.281049 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:27:55.281189 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:27:55.281235 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 04:27:55.281336 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 04:27:55.281461 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1202,"mutex_wait_us":180,"thread_start_us":100,"threads_started":1}
I20260620 04:27:55.283620 16266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:55.275299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34337 (local address 127.0.0.1:38096)
0620 04:27:55.276668 (+ 1369us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:55.276678 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 04:27:55.276753 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:27:55.278486 (+ 1733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:27:55.278489 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:27:55.278507 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:27:55.278694 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:27:55.278702 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:27:55.280032 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:27:55.280034 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:27:55.281099 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:27:55.281110 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:27:55.283239 (+ 2129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:27:55.283243 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:27:55.283329 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:27:55.283353 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 04:27:55.283428 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 04:27:55.283494 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1327,"mutex_wait_us":70,"thread_start_us":92,"threads_started":1}
I20260620 04:27:55.283849 16270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:55.275867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38096 (local address 127.0.98.1:34337)
0620 04:27:55.276576 (+ 709us) server_negotiation.cc:206] Beginning negotiation
0620 04:27:55.276585 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 04:27:55.276763 (+ 178us) server_negotiation.cc:407] Connection header received
0620 04:27:55.276845 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:27:55.276850 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:27:55.277037 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:27:55.277190 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:27:55.279072 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:55.279875 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:55.281238 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:55.283112 (+ 1874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:55.283215 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:27:55.283344 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 04:27:55.283460 (+ 116us) server_negotiation.cc:299] Negotiation successful
0620 04:27:55.283545 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":631,"thread_start_us":119,"threads_started":1}
I20260620 04:27:55.284879 16189 tablet_service.cc:1467] Processing CreateTablet for tablet bd6f7e2efef74b1b9026d3d982f0ee83 (DEFAULT_TABLE table=test-table [id=91aaa862b6a648a885dedeabfa907fb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:27:55.284888 16055 tablet_service.cc:1467] Processing CreateTablet for tablet bd6f7e2efef74b1b9026d3d982f0ee83 (DEFAULT_TABLE table=test-table [id=91aaa862b6a648a885dedeabfa907fb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:27:55.285838 16189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd6f7e2efef74b1b9026d3d982f0ee83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:27:55.285880 16055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd6f7e2efef74b1b9026d3d982f0ee83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:27:55.286386 15921 tablet_service.cc:1467] Processing CreateTablet for tablet bd6f7e2efef74b1b9026d3d982f0ee83 (DEFAULT_TABLE table=test-table [id=91aaa862b6a648a885dedeabfa907fb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:27:55.287282 15921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd6f7e2efef74b1b9026d3d982f0ee83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:27:55.291837 16272 tablet_bootstrap.cc:492] T bd6f7e2efef74b1b9026d3d982f0ee83 P ce2d44319f6d48ee894f1d5bfc293a78: Bootstrap starting.
I20260620 04:27:55.294027 16274 tablet_bootstrap.cc:492] T bd6f7e2efef74b1b9026d3d982f0ee83 P 54e13c6e85e747aea02cf40981ed4499: Bootstrap starting.
I20260620 04:27:55.294147 16272 tablet_bootstrap.cc:654] T bd6f7e2efef74b1b9026d3d982f0ee83 P ce2d44319f6d48ee894f1d5bfc293a78: Neither blocks nor log segments found. Creating new log.
I20260620 04:27:55.294773 16273 tablet_bootstrap.cc:492] T bd6f7e2efef74b1b9026d3d982f0ee83 P a546aa48667e483bbbde10dfcdf180ff: Bootstrap starting.
I20260620 04:27:55.295413 16272 log.cc:824] T bd6f7e2efef74b1b9026d3d982f0ee83 P ce2d44319f6d48ee894f1d5bfc293a78: Log is configured to *not* fsync() on all Append() calls
I20260620 04:27:55.296185 16274 tablet_bootstrap.cc:654] T bd6f7e2efef74b1b9026d3d982f0ee83 P 54e13c6e85e747aea02cf40981ed4499: Neither blocks nor log segments found. Creating new log.
I20260620 04:27:55.297276 16273 tablet_bootstrap.cc:654] T bd6f7e2efef74b1b9026d3d982f0ee83 P a546aa48667e483bbbde10dfcdf180ff: Neither blocks nor log segments found. Creating new log.
I20260620 04:27:55.297396 16274 log.cc:824] T bd6f7e2efef74b1b9026d3d982f0ee83 P 54e13c6e85e747aea02cf40981ed4499: Log is configured to *not* fsync() on all Append() calls
I20260620 04:27:55.297577 16272 tablet_bootstrap.cc:492] T bd6f7e2efef74b1b9026d3d982f0ee83 P ce2d44319f6d48ee894f1d5bfc293a78: No bootstrap required, opened a new log
I20260620 04:27:55.297758 16272 ts_tablet_manager.cc:1397] T bd6f7e2efef74b1b9026d3d982f0ee83 P ce2d44319f6d48ee894f1d5bfc293a78: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 04:27:55.297995 16273 log.cc:824] T bd6f7e2efef74b1b9026d3d982f0ee83 P a546aa48667e483bbbde10dfcdf180ff: Log is configured to *not* fsync() on all Append() calls
I20260620 04:27:55.299233 16274 tablet_bootstrap.cc:492] T bd6f7e2efef74b1b9026d3d982f0ee83 P 54e13c6e85e747aea02cf40981ed4499: No bootstrap required, opened a new log
I20260620 04:27:55.299407 16274 ts_tablet_manager.cc:1397] T bd6f7e2efef74b1b9026d3d982f0ee83 P 54e13c6e85e747aea02cf40981ed4499: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 04:27:55.299769 16273 tablet_bootstrap.cc:492] T bd6f7e2efef74b1b9026d3d982f0ee83 P a546aa48667e483bbbde10dfcdf180ff: No bootstrap required, opened a new log
I20260620 04:27:55.299930 16273 ts_tablet_manager.cc:1397] T bd6f7e2efef74b1b9026d3d982f0ee83 P a546aa48667e483bbbde10dfcdf180ff: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 04:27:55.300815 16272 raft_consensus.cc:348] T bd6f7e2efef74b1b9026d3d982f0ee83 P ce2d44319f6d48ee894f1d5bfc293a78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a546aa48667e483bbbde10dfcdf180ff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38671 } } peers { permanent_uuid: "ce2d44319f6d48ee894f1d5bfc293a78" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39419 } } peers { permanent_uuid: "54e13c6e85e747aea02cf40981ed4499" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34337 } }
I20260620 04:27:55.301028 16272 raft_consensus.cc:374] T bd6f7e2efef74b1b9026d3d982f0ee83 P ce2d44319f6d48ee894f1d5bfc293a78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:27:55.301096 16272 raft_consensus.cc:729] T bd6f7e2efef74b1b9026d3d982f0ee83 P ce2d44319f6d48ee894f1d5bfc293a78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce2d44319f6d48ee894f1d5bfc293a78, State: Initialized, Role: FOLLOWER
I20260620 04:27:55.301503 16272 consensus_queue.cc:260] T bd6f7e2efef74b1b9026d3d982f0ee83 P ce2d44319f6d48ee894f1d5bfc293a78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a546aa48667e483bbbde10dfcdf180ff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38671 } } peers { permanent_uuid: "ce2d44319f6d48ee894f1d5bfc293a78" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39419 } } peers { permanent_uuid: "54e13c6e85e747aea02cf40981ed4499" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34337 } }
I20260620 04:27:55.302078 16255 heartbeater.cc:503] Master 127.0.98.62:39499 was elected leader, sending a full tablet report...
I20260620 04:27:55.302408 16272 ts_tablet_manager.cc:1428] T bd6f7e2efef74b1b9026d3d982f0ee83 P ce2d44319f6d48ee894f1d5bfc293a78: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 04:27:55.303121 16274 raft_consensus.cc:348] T bd6f7e2efef74b1b9026d3d982f0ee83 P 54e13c6e85e747aea02cf40981ed4499 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a546aa48667e483bbbde10dfcdf180ff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38671 } } peers { permanent_uuid: "ce2d44319f6d48ee894f1d5bfc293a78" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39419 } } peers { permanent_uuid: "54e13c6e85e747aea02cf40981ed4499" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34337 } }
I20260620 04:27:55.303396 16274 raft_consensus.cc:374] T bd6f7e2efef74b1b9026d3d982f0ee83 P 54e13c6e85e747aea02cf40981ed4499 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:27:55.303458 16274 raft_consensus.cc:729] T bd6f7e2efef74b1b9026d3d982f0ee83 P 54e13c6e85e747aea02cf40981ed4499 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54e13c6e85e747aea02cf40981ed4499, State: Initialized, Role: FOLLOWER
I20260620 04:27:55.303859 16274 consensus_queue.cc:260] T bd6f7e2efef74b1b9026d3d982f0ee83 P 54e13c6e85e747aea02cf40981ed4499 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a546aa48667e483bbbde10dfcdf180ff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38671 } } peers { permanent_uuid: "ce2d44319f6d48ee894f1d5bfc293a78" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39419 } } peers { permanent_uuid: "54e13c6e85e747aea02cf40981ed4499" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34337 } }
I20260620 04:27:55.304550 15987 heartbeater.cc:503] Master 127.0.98.62:39499 was elected leader, sending a full tablet report...
I20260620 04:27:55.304459 16273 raft_consensus.cc:348] T bd6f7e2efef74b1b9026d3d982f0ee83 P a546aa48667e483bbbde10dfcdf180ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a546aa48667e483bbbde10dfcdf180ff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38671 } } peers { permanent_uuid: "ce2d44319f6d48ee894f1d5bfc293a78" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39419 } } peers { permanent_uuid: "54e13c6e85e747aea02cf40981ed4499" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34337 } }
I20260620 04:27:55.304661 16273 raft_consensus.cc:374] T bd6f7e2efef74b1b9026d3d982f0ee83 P a546aa48667e483bbbde10dfcdf180ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:27:55.304690 16273 raft_consensus.cc:729] T bd6f7e2efef74b1b9026d3d982f0ee83 P a546aa48667e483bbbde10dfcdf180ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a546aa48667e483bbbde10dfcdf180ff, State: Initialized, Role: FOLLOWER
I20260620 04:27:55.305030 16273 consensus_queue.cc:260] T bd6f7e2efef74b1b9026d3d982f0ee83 P a546aa48667e483bbbde10dfcdf180ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a546aa48667e483bbbde10dfcdf180ff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38671 } } peers { permanent_uuid: "ce2d44319f6d48ee894f1d5bfc293a78" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39419 } } peers { permanent_uuid: "54e13c6e85e747aea02cf40981ed4499" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34337 } }
I20260620 04:27:55.305621 16121 heartbeater.cc:503] Master 127.0.98.62:39499 was elected leader, sending a full tablet report...
I20260620 04:27:55.305917 16274 ts_tablet_manager.cc:1428] T bd6f7e2efef74b1b9026d3d982f0ee83 P 54e13c6e85e747aea02cf40981ed4499: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 04:27:55.305922 16273 ts_tablet_manager.cc:1428] T bd6f7e2efef74b1b9026d3d982f0ee83 P a546aa48667e483bbbde10dfcdf180ff: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
W20260620 04:27:55.322657 15988 tablet.cc:2369] T bd6f7e2efef74b1b9026d3d982f0ee83 P 54e13c6e85e747aea02cf40981ed4499: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 04:27:55.485502 16256 tablet.cc:2369] T bd6f7e2efef74b1b9026d3d982f0ee83 P ce2d44319f6d48ee894f1d5bfc293a78: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 04:27:55.525506 16122 tablet.cc:2369] T bd6f7e2efef74b1b9026d3d982f0ee83 P a546aa48667e483bbbde10dfcdf180ff: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 04:27:55.673120 16280 raft_consensus.cc:482] T bd6f7e2efef74b1b9026d3d982f0ee83 P a546aa48667e483bbbde10dfcdf180ff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:27:55.673324 16280 raft_consensus.cc:504] T bd6f7e2efef74b1b9026d3d982f0ee83 P a546aa48667e483bbbde10dfcdf180ff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a546aa48667e483bbbde10dfcdf180ff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38671 } } peers { permanent_uuid: "ce2d44319f6d48ee894f1d5bfc293a78" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39419 } } peers { permanent_uuid: "54e13c6e85e747aea02cf40981ed4499" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34337 } }
I20260620 04:27:55.674489 16280 leader_election.cc:288] T bd6f7e2efef74b1b9026d3d982f0ee83 P a546aa48667e483bbbde10dfcdf180ff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ce2d44319f6d48ee894f1d5bfc293a78 (127.0.98.3:39419), 54e13c6e85e747aea02cf40981ed4499 (127.0.98.1:34337)
I20260620 04:27:55.678181 16271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:55.674808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52169 (local address 127.0.98.3:39419)
0620 04:27:55.674932 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 04:27:55.674936 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:27:55.675133 (+ 197us) server_negotiation.cc:407] Connection header received
0620 04:27:55.675251 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:27:55.675254 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:27:55.675308 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:27:55.675375 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:27:55.675652 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:55.676211 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:55.677264 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:55.677661 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:55.677748 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:27:55.677811 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 04:27:55.677986 (+ 175us) server_negotiation.cc:299] Negotiation successful
0620 04:27:55.678053 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 04:27:55.678201 16270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:55.674863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56833 (local address 127.0.98.1:34337)
0620 04:27:55.674983 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 04:27:55.674987 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:27:55.675244 (+ 257us) server_negotiation.cc:407] Connection header received
0620 04:27:55.675387 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:27:55.675390 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:27:55.675444 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:27:55.675518 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:27:55.675763 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:55.676331 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:55.677303 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:55.677673 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:55.677766 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:27:55.677826 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 04:27:55.678025 (+ 199us) server_negotiation.cc:299] Negotiation successful
0620 04:27:55.678089 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 04:27:55.678205 16281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:55.674749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34337 (local address 127.0.98.2:56833)
0620 04:27:55.675144 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:55.675156 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 04:27:55.675284 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:27:55.675532 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:27:55.675534 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:27:55.675548 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:27:55.675661 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:27:55.675667 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:27:55.676436 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:27:55.676439 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:27:55.677182 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:27:55.677190 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:27:55.677780 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:27:55.677783 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:27:55.677916 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:27:55.677932 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 04:27:55.678003 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 04:27:55.678062 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"mutex_wait_us":35,"thread_start_us":75,"threads_started":1}
I20260620 04:27:55.678213 16282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:55.674736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39419 (local address 127.0.98.2:52169)
0620 04:27:55.675033 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:55.675044 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 04:27:55.675151 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:27:55.675402 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:27:55.675405 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:27:55.675421 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:27:55.675549 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:27:55.675556 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:27:55.676311 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:27:55.676314 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:27:55.677148 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:27:55.677157 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:27:55.677772 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:27:55.677776 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:27:55.677885 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:27:55.677904 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 04:27:55.677960 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 04:27:55.678024 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"mutex_wait_us":3,"thread_start_us":69,"threads_started":1}
I20260620 04:27:55.678803 15941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd6f7e2efef74b1b9026d3d982f0ee83" candidate_uuid: "a546aa48667e483bbbde10dfcdf180ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54e13c6e85e747aea02cf40981ed4499" is_pre_election: true
I20260620 04:27:55.678910 16209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd6f7e2efef74b1b9026d3d982f0ee83" candidate_uuid: "a546aa48667e483bbbde10dfcdf180ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce2d44319f6d48ee894f1d5bfc293a78" is_pre_election: true
I20260620 04:27:55.679096 15941 raft_consensus.cc:2454] T bd6f7e2efef74b1b9026d3d982f0ee83 P 54e13c6e85e747aea02cf40981ed4499 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a546aa48667e483bbbde10dfcdf180ff in term 0.
I20260620 04:27:55.679189 16209 raft_consensus.cc:2454] T bd6f7e2efef74b1b9026d3d982f0ee83 P ce2d44319f6d48ee894f1d5bfc293a78 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a546aa48667e483bbbde10dfcdf180ff in term 0.
I20260620 04:27:55.679606 16010 leader_election.cc:302] T bd6f7e2efef74b1b9026d3d982f0ee83 P a546aa48667e483bbbde10dfcdf180ff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54e13c6e85e747aea02cf40981ed4499, a546aa48667e483bbbde10dfcdf180ff; no voters:
I20260620 04:27:55.679853 16280 raft_consensus.cc:2781] T bd6f7e2efef74b1b9026d3d982f0ee83 P a546aa48667e483bbbde10dfcdf180ff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 04:27:55.679935 16280 raft_consensus.cc:482] T bd6f7e2efef74b1b9026d3d982f0ee83 P a546aa48667e483bbbde10dfcdf180ff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 04:27:55.680006 16280 raft_consensus.cc:3037] T bd6f7e2efef74b1b9026d3d982f0ee83 P a546aa48667e483bbbde10dfcdf180ff [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:27:55.680954 16280 raft_consensus.cc:504] T bd6f7e2efef74b1b9026d3d982f0ee83 P a546aa48667e483bbbde10dfcdf180ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a546aa48667e483bbbde10dfcdf180ff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38671 } } peers { permanent_uuid: "ce2d44319f6d48ee894f1d5bfc293a78" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39419 } } peers { permanent_uuid: "54e13c6e85e747aea02cf40981ed4499" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34337 } }
I20260620 04:27:55.681239 16280 leader_election.cc:288] T bd6f7e2efef74b1b9026d3d982f0ee83 P a546aa48667e483bbbde10dfcdf180ff [CANDIDATE]: Term 1 election: Requested vote from peers ce2d44319f6d48ee894f1d5bfc293a78 (127.0.98.3:39419), 54e13c6e85e747aea02cf40981ed4499 (127.0.98.1:34337)
I20260620 04:27:55.681519 16209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd6f7e2efef74b1b9026d3d982f0ee83" candidate_uuid: "a546aa48667e483bbbde10dfcdf180ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce2d44319f6d48ee894f1d5bfc293a78"
I20260620 04:27:55.681615 16209 raft_consensus.cc:3037] T bd6f7e2efef74b1b9026d3d982f0ee83 P ce2d44319f6d48ee894f1d5bfc293a78 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:27:55.681620 15941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd6f7e2efef74b1b9026d3d982f0ee83" candidate_uuid: "a546aa48667e483bbbde10dfcdf180ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54e13c6e85e747aea02cf40981ed4499"
I20260620 04:27:55.681730 15941 raft_consensus.cc:3037] T bd6f7e2efef74b1b9026d3d982f0ee83 P 54e13c6e85e747aea02cf40981ed4499 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:27:55.682579 16209 raft_consensus.cc:2454] T bd6f7e2efef74b1b9026d3d982f0ee83 P ce2d44319f6d48ee894f1d5bfc293a78 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a546aa48667e483bbbde10dfcdf180ff in term 1.
I20260620 04:27:55.682633 15941 raft_consensus.cc:2454] T bd6f7e2efef74b1b9026d3d982f0ee83 P 54e13c6e85e747aea02cf40981ed4499 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a546aa48667e483bbbde10dfcdf180ff in term 1.
I20260620 04:27:55.682857 16009 leader_election.cc:302] T bd6f7e2efef74b1b9026d3d982f0ee83 P a546aa48667e483bbbde10dfcdf180ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a546aa48667e483bbbde10dfcdf180ff, ce2d44319f6d48ee894f1d5bfc293a78; no voters:
I20260620 04:27:55.683003 16280 raft_consensus.cc:2781] T bd6f7e2efef74b1b9026d3d982f0ee83 P a546aa48667e483bbbde10dfcdf180ff [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:27:55.683209 16280 raft_consensus.cc:686] T bd6f7e2efef74b1b9026d3d982f0ee83 P a546aa48667e483bbbde10dfcdf180ff [term 1 LEADER]: Becoming Leader. State: Replica: a546aa48667e483bbbde10dfcdf180ff, State: Running, Role: LEADER
I20260620 04:27:55.683550 16280 consensus_queue.cc:237] T bd6f7e2efef74b1b9026d3d982f0ee83 P a546aa48667e483bbbde10dfcdf180ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a546aa48667e483bbbde10dfcdf180ff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38671 } } peers { permanent_uuid: "ce2d44319f6d48ee894f1d5bfc293a78" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39419 } } peers { permanent_uuid: "54e13c6e85e747aea02cf40981ed4499" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34337 } }
I20260620 04:27:55.686522 15796 catalog_manager.cc:5495] T bd6f7e2efef74b1b9026d3d982f0ee83 P a546aa48667e483bbbde10dfcdf180ff reported cstate change: term changed from 0 to 1, leader changed from <none> to a546aa48667e483bbbde10dfcdf180ff (127.0.98.2). New cstate: current_term: 1 leader_uuid: "a546aa48667e483bbbde10dfcdf180ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a546aa48667e483bbbde10dfcdf180ff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce2d44319f6d48ee894f1d5bfc293a78" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39419 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54e13c6e85e747aea02cf40981ed4499" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34337 } health_report { overall_health: UNKNOWN } } }
I20260620 04:27:55.730247 16269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:55.726615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33336 (local address 127.0.98.2:38671)
0620 04:27:55.726748 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 04:27:55.726753 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:27:55.726768 (+ 15us) server_negotiation.cc:407] Connection header received
0620 04:27:55.726885 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:27:55.726887 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:27:55.726943 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:27:55.727036 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:27:55.727298 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:55.727790 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:55.728654 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:27:55.728859 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:27:55.728943 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:27:55.729215 (+ 272us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:27:55.729643 (+ 428us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:27:55.730015 (+ 372us) server_negotiation.cc:1035] Waiting for connection context
0620 04:27:55.730049 (+ 34us) server_negotiation.cc:299] Negotiation successful
0620 04:27:55.730106 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 04:27:55.736105 16209 raft_consensus.cc:1261] T bd6f7e2efef74b1b9026d3d982f0ee83 P ce2d44319f6d48ee894f1d5bfc293a78 [term 1 FOLLOWER]: Refusing update from remote peer a546aa48667e483bbbde10dfcdf180ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:27:55.736105 15941 raft_consensus.cc:1261] T bd6f7e2efef74b1b9026d3d982f0ee83 P 54e13c6e85e747aea02cf40981ed4499 [term 1 FOLLOWER]: Refusing update from remote peer a546aa48667e483bbbde10dfcdf180ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:27:55.736804 16283 consensus_queue.cc:1035] T bd6f7e2efef74b1b9026d3d982f0ee83 P a546aa48667e483bbbde10dfcdf180ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "54e13c6e85e747aea02cf40981ed4499" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:27:55.736963 16280 consensus_queue.cc:1035] T bd6f7e2efef74b1b9026d3d982f0ee83 P a546aa48667e483bbbde10dfcdf180ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce2d44319f6d48ee894f1d5bfc293a78" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39419 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:27:55.745139 16287 mvcc.cc:205] Tried to move back new op lower bound from 7298783951807389696 to 7298783951602270208. Current Snapshot: MvccSnapshot[applied={T|T < 7298783951807389696}]
I20260620 04:27:55.753697 15796 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56648:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:27:55.753885 15796 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56648:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:27:55.756345 15796 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cccb8275856c44bbb18eacd620970116: Sending DeleteTablet for 3 replicas of tablet bd6f7e2efef74b1b9026d3d982f0ee83
I20260620 04:27:55.757053 16055 tablet_service.cc:1514] Processing DeleteTablet for tablet bd6f7e2efef74b1b9026d3d982f0ee83 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:27:55 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33322
I20260620 04:27:55.757084 16189 tablet_service.cc:1514] Processing DeleteTablet for tablet bd6f7e2efef74b1b9026d3d982f0ee83 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:27:55 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43480
I20260620 04:27:55.757145 15921 tablet_service.cc:1514] Processing DeleteTablet for tablet bd6f7e2efef74b1b9026d3d982f0ee83 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:27:55 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38096
I20260620 04:27:55.757262 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 15858
I20260620 04:27:55.761857 16297 tablet_replica.cc:330] T bd6f7e2efef74b1b9026d3d982f0ee83 P a546aa48667e483bbbde10dfcdf180ff: stopping tablet replica
I20260620 04:27:55.762176 16296 tablet_replica.cc:330] T bd6f7e2efef74b1b9026d3d982f0ee83 P ce2d44319f6d48ee894f1d5bfc293a78: stopping tablet replica
I20260620 04:27:55.762207 16297 raft_consensus.cc:2229] T bd6f7e2efef74b1b9026d3d982f0ee83 P a546aa48667e483bbbde10dfcdf180ff [term 1 LEADER]: Raft consensus shutting down.
I20260620 04:27:55.762547 16297 raft_consensus.cc:2258] T bd6f7e2efef74b1b9026d3d982f0ee83 P a546aa48667e483bbbde10dfcdf180ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:27:55.762593 16296 raft_consensus.cc:2229] T bd6f7e2efef74b1b9026d3d982f0ee83 P ce2d44319f6d48ee894f1d5bfc293a78 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:27:55.763326 16296 raft_consensus.cc:2258] T bd6f7e2efef74b1b9026d3d982f0ee83 P ce2d44319f6d48ee894f1d5bfc293a78 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260620 04:27:55.763867 15782 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34337 (error 108)
I20260620 04:27:55.764019 16297 ts_tablet_manager.cc:1905] T bd6f7e2efef74b1b9026d3d982f0ee83 P a546aa48667e483bbbde10dfcdf180ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:27:55.764627 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 15992
I20260620 04:27:55.765800 16266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:55.765535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34337 (local address 127.0.0.1:38112)
0620 04:27:55.765641 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:55.765714 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260620 04:27:55.766072 15782 catalog_manager.cc:4584] TS 54e13c6e85e747aea02cf40981ed4499 (127.0.98.1:34337): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bd6f7e2efef74b1b9026d3d982f0ee83: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34337: connect: Connection refused (error 111)
I20260620 04:27:55.766372 16296 ts_tablet_manager.cc:1905] T bd6f7e2efef74b1b9026d3d982f0ee83 P ce2d44319f6d48ee894f1d5bfc293a78: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:27:55.768528 16296 ts_tablet_manager.cc:1918] T bd6f7e2efef74b1b9026d3d982f0ee83 P ce2d44319f6d48ee894f1d5bfc293a78: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:27:55.768612 16296 log.cc:1196] T bd6f7e2efef74b1b9026d3d982f0ee83 P ce2d44319f6d48ee894f1d5bfc293a78: Deleting WAL directory at /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-2/wal/wals/bd6f7e2efef74b1b9026d3d982f0ee83
I20260620 04:27:55.768852 16296 ts_tablet_manager.cc:1939] T bd6f7e2efef74b1b9026d3d982f0ee83 P ce2d44319f6d48ee894f1d5bfc293a78: Deleting consensus metadata
I20260620 04:27:55.770030 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 16126
I20260620 04:27:55.770856 16266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:27:55.770269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38671 (local address 127.0.0.1:33342)
0620 04:27:55.770723 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0620 04:27:55.770775 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:38671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368}
I20260620 04:27:55.774828 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 15764
I20260620 04:28:15.780169 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39499
--webserver_interface=127.0.98.62
--webserver_port=38563
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39499
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:28:15.885880 16299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:28:15.886140 16299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:28:15.886193 16299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:28:15.890161 16299 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 04:28:15.890215 16299 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 04:28:15.890235 16299 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 04:28:15.890254 16299 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:28:15.890276 16299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:28:15.890306 16299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:28:15.890336 16299 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:28:15.890363 16299 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:28:15.895493 16299 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39499
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39499
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=38563
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16299
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:28:15.896535 16299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:28:15.897509 16299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:28:15.909632 16306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:28:15.909655 16308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:28:15.909742 16299 server_base.cc:1034] running on GCE node
W20260620 04:28:15.909632 16305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:28:15.910521 16299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:28:15.911602 16299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:28:15.912763 16299 hybrid_clock.cc:648] HybridClock initialized: now 1781929695912735 us; error 41 us; skew 500 ppm
Jun 20 04:28:15 dist-test-slave-nk6d krb5kdc[15751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929695, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:28:15.915514 16299 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 04:28:15.916407 16299 webserver.cc:460] Webserver started at http://127.0.98.62:38563/ using document root <none> and password file <none>
I20260620 04:28:15.916954 16299 fs_manager.cc:359] Metadata directory not provided
I20260620 04:28:15.917022 16299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:28:15.920703 16299 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:28:15.922847 16315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:28:15.924054 16299 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:28:15.924163 16299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/master-0/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/master-0/wal
uuid: "cccb8275856c44bbb18eacd620970116"
format_stamp: "Formatted at 2026-06-20 04:27:54 on dist-test-slave-nk6d"
server_key: "130fb43b0ecebb08308b5116486efb34"
server_key_iv: "fcf121ace544218ffbc10f663267120a"
server_key_version: "encryptionkey@0"
I20260620 04:28:15.924724 16299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:28:15.937402 16299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:28:15.940083 16299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:28:15.940248 16299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:28:15.946681 16299 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39499
I20260620 04:28:15.946918 16367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39499 every 8 connection(s)
I20260620 04:28:15.947710 16299 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:28:15.951870 16368 sys_catalog.cc:263] Verifying existing consensus state
I20260620 04:28:15.953464 16368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cccb8275856c44bbb18eacd620970116: Bootstrap starting.
I20260620 04:28:15.956104 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 16299
I20260620 04:28:15.956494 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:34337
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:39499
--webserver_port=44573
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 04:28:15.970204 16368 log.cc:824] T 00000000000000000000000000000000 P cccb8275856c44bbb18eacd620970116: Log is configured to *not* fsync() on all Append() calls
I20260620 04:28:15.978783 16368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cccb8275856c44bbb18eacd620970116: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 04:28:15.979113 16368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cccb8275856c44bbb18eacd620970116: Bootstrap complete.
I20260620 04:28:15.982906 16368 raft_consensus.cc:348] T 00000000000000000000000000000000 P cccb8275856c44bbb18eacd620970116 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cccb8275856c44bbb18eacd620970116" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39499 } }
I20260620 04:28:15.983283 16368 raft_consensus.cc:729] T 00000000000000000000000000000000 P cccb8275856c44bbb18eacd620970116 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cccb8275856c44bbb18eacd620970116, State: Initialized, Role: FOLLOWER
I20260620 04:28:15.983765 16368 consensus_queue.cc:260] T 00000000000000000000000000000000 P cccb8275856c44bbb18eacd620970116 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cccb8275856c44bbb18eacd620970116" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39499 } }
I20260620 04:28:15.983879 16368 raft_consensus.cc:388] T 00000000000000000000000000000000 P cccb8275856c44bbb18eacd620970116 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:28:15.983945 16368 raft_consensus.cc:482] T 00000000000000000000000000000000 P cccb8275856c44bbb18eacd620970116 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:28:15.984061 16368 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cccb8275856c44bbb18eacd620970116 [term 1 FOLLOWER]: Advancing to term 2
I20260620 04:28:15.985965 16368 raft_consensus.cc:504] T 00000000000000000000000000000000 P cccb8275856c44bbb18eacd620970116 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cccb8275856c44bbb18eacd620970116" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39499 } }
I20260620 04:28:15.986280 16368 leader_election.cc:302] T 00000000000000000000000000000000 P cccb8275856c44bbb18eacd620970116 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cccb8275856c44bbb18eacd620970116; no voters:
I20260620 04:28:15.986563 16368 leader_election.cc:288] T 00000000000000000000000000000000 P cccb8275856c44bbb18eacd620970116 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 04:28:15.986657 16372 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cccb8275856c44bbb18eacd620970116 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 04:28:15.986851 16372 raft_consensus.cc:686] T 00000000000000000000000000000000 P cccb8275856c44bbb18eacd620970116 [term 2 LEADER]: Becoming Leader. State: Replica: cccb8275856c44bbb18eacd620970116, State: Running, Role: LEADER
I20260620 04:28:15.987150 16372 consensus_queue.cc:237] T 00000000000000000000000000000000 P cccb8275856c44bbb18eacd620970116 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cccb8275856c44bbb18eacd620970116" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39499 } }
I20260620 04:28:15.987805 16368 sys_catalog.cc:564] T 00000000000000000000000000000000 P cccb8275856c44bbb18eacd620970116 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:28:15.988119 16372 sys_catalog.cc:455] T 00000000000000000000000000000000 P cccb8275856c44bbb18eacd620970116 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cccb8275856c44bbb18eacd620970116. Latest consensus state: current_term: 2 leader_uuid: "cccb8275856c44bbb18eacd620970116" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cccb8275856c44bbb18eacd620970116" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39499 } } }
I20260620 04:28:15.988230 16372 sys_catalog.cc:458] T 00000000000000000000000000000000 P cccb8275856c44bbb18eacd620970116 [sys.catalog]: This master's current role is: LEADER
I20260620 04:28:15.988612 16376 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:28:15.988619 16373 sys_catalog.cc:455] T 00000000000000000000000000000000 P cccb8275856c44bbb18eacd620970116 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "cccb8275856c44bbb18eacd620970116" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cccb8275856c44bbb18eacd620970116" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39499 } } }
I20260620 04:28:15.988871 16373 sys_catalog.cc:458] T 00000000000000000000000000000000 P cccb8275856c44bbb18eacd620970116 [sys.catalog]: This master's current role is: LEADER
I20260620 04:28:15.993230 16376 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:28:15.993841 16376 catalog_manager.cc:1259] Loaded cluster ID: 6722e5d138fc44cebacf726ef2c97bb5
I20260620 04:28:15.993901 16376 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:28:15.996351 16376 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:28:15.997036 16376 catalog_manager.cc:5879] T 00000000000000000000000000000000 P cccb8275856c44bbb18eacd620970116: Loaded TSK: 0
I20260620 04:28:16.009068 16376 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cccb8275856c44bbb18eacd620970116: Generated new TSK 1
I20260620 04:28:16.009359 16376 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 04:28:16.068687 16370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:28:16.068964 16370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:28:16.069026 16370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:28:16.073120 16370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:28:16.073186 16370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:28:16.073272 16370 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 04:28:16.078656 16370 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:34337
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=44573
--tserver_master_addrs=127.0.98.62:39499
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16370
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:28:16.079689 16370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:28:16.080729 16370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:28:16.090957 16398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:28:16.090993 16395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:28:16.091101 16370 server_base.cc:1034] running on GCE node
W20260620 04:28:16.091269 16396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:28:16.091728 16370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:28:16.092415 16370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:28:16.093572 16370 hybrid_clock.cc:648] HybridClock initialized: now 1781929696093545 us; error 49 us; skew 500 ppm
Jun 20 04:28:16 dist-test-slave-nk6d krb5kdc[15751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929696, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:28:16.096335 16370 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 04:28:16.097216 16370 webserver.cc:460] Webserver started at http://127.0.98.1:44573/ using document root <none> and password file <none>
I20260620 04:28:16.097764 16370 fs_manager.cc:359] Metadata directory not provided
I20260620 04:28:16.097843 16370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:28:16.101840 16370 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 04:28:16.104064 16405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:28:16.105310 16370 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:28:16.105402 16370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-0/wal
uuid: "54e13c6e85e747aea02cf40981ed4499"
format_stamp: "Formatted at 2026-06-20 04:27:54 on dist-test-slave-nk6d"
server_key: "cf70bf115049eb5d5d33ad49fef6187c"
server_key_iv: "b6361780b97fbbe09e78d86cb838f684"
server_key_version: "encryptionkey@0"
I20260620 04:28:16.106009 16370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:28:16.131527 16370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:28:16.134300 16370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:28:16.134450 16370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:28:16.134989 16370 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:28:16.136005 16412 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 04:28:16.139945 16370 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 04:28:16.140043 16370 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260620 04:28:16.140096 16370 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 04:28:16.141701 16370 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 04:28:16.141752 16370 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260620 04:28:16.142040 16412 tablet_bootstrap.cc:492] T bd6f7e2efef74b1b9026d3d982f0ee83 P 54e13c6e85e747aea02cf40981ed4499: Bootstrap starting.
I20260620 04:28:16.150480 16370 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34337
I20260620 04:28:16.150564 16519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34337 every 8 connection(s)
I20260620 04:28:16.151391 16370 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-0/data/info.pb
I20260620 04:28:16.152637 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 16370
I20260620 04:28:16.153067 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:38671
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:39499
--webserver_port=43673
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 04:28:16.160815 16412 log.cc:824] T bd6f7e2efef74b1b9026d3d982f0ee83 P 54e13c6e85e747aea02cf40981ed4499: Log is configured to *not* fsync() on all Append() calls
Jun 20 04:28:16 dist-test-slave-nk6d krb5kdc[15751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929696, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:28:16.168893 16412 tablet_bootstrap.cc:492] T bd6f7e2efef74b1b9026d3d982f0ee83 P 54e13c6e85e747aea02cf40981ed4499: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 04:28:16.169322 16412 tablet_bootstrap.cc:492] T bd6f7e2efef74b1b9026d3d982f0ee83 P 54e13c6e85e747aea02cf40981ed4499: Bootstrap complete.
I20260620 04:28:16.170400 16412 ts_tablet_manager.cc:1397] T bd6f7e2efef74b1b9026d3d982f0ee83 P 54e13c6e85e747aea02cf40981ed4499: Time spent bootstrapping tablet: real 0.029s user 0.018s sys 0.008s
I20260620 04:28:16.171525 16524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:16.154707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40935 (local address 127.0.98.62:39499)
0620 04:28:16.155149 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0620 04:28:16.155167 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 04:28:16.155278 (+ 111us) server_negotiation.cc:407] Connection header received
0620 04:28:16.156762 (+ 1484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:28:16.156781 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:28:16.157205 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:28:16.157583 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:28:16.159141 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:28:16.160706 (+ 1565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:28:16.162041 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:28:16.162771 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:28:16.162942 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:28:16.165884 (+ 2942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:28:16.165891 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:28:16.165907 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:28:16.165946 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:28:16.168001 (+ 2055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:28:16.168588 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:28:16.168594 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:28:16.168601 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:28:16.168759 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:28:16.169146 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:28:16.169150 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:28:16.169152 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:28:16.169613 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:28:16.169737 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 04:28:16.170348 (+ 611us) server_negotiation.cc:299] Negotiation successful
0620 04:28:16.170658 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":105,"threads_started":1}
I20260620 04:28:16.171542 16522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:16.153718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39499 (local address 127.0.98.1:40935)
0620 04:28:16.155109 (+ 1391us) negotiation.cc:107] Waiting for socket to connect
0620 04:28:16.155165 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 04:28:16.156261 (+ 1096us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:28:16.157695 (+ 1434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:28:16.157704 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:28:16.158199 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:28:16.158944 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:28:16.158966 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:28:16.160878 (+ 1912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:28:16.160881 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:28:16.161913 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:28:16.161923 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:28:16.162920 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:28:16.162924 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:28:16.163104 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:28:16.163758 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:28:16.163789 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:28:16.165630 (+ 1841us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:28:16.168118 (+ 2488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:28:16.168125 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:28:16.168139 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:28:16.168469 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:28:16.168862 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:28:16.168865 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:28:16.168867 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:28:16.169042 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:28:16.169728 (+ 686us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:28:16.169735 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:28:16.170034 (+ 299us) client_negotiation.cc:769] Sending connection context
0620 04:28:16.170223 (+ 189us) client_negotiation.cc:240] Negotiation successful
0620 04:28:16.170616 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1159,"thread_start_us":132,"threads_started":1}
I20260620 04:28:16.173310 16412 raft_consensus.cc:348] T bd6f7e2efef74b1b9026d3d982f0ee83 P 54e13c6e85e747aea02cf40981ed4499 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a546aa48667e483bbbde10dfcdf180ff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38671 } } peers { permanent_uuid: "ce2d44319f6d48ee894f1d5bfc293a78" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39419 } } peers { permanent_uuid: "54e13c6e85e747aea02cf40981ed4499" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34337 } }
I20260620 04:28:16.173769 16412 raft_consensus.cc:729] T bd6f7e2efef74b1b9026d3d982f0ee83 P 54e13c6e85e747aea02cf40981ed4499 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54e13c6e85e747aea02cf40981ed4499, State: Initialized, Role: FOLLOWER
I20260620 04:28:16.174252 16412 consensus_queue.cc:260] T bd6f7e2efef74b1b9026d3d982f0ee83 P 54e13c6e85e747aea02cf40981ed4499 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a546aa48667e483bbbde10dfcdf180ff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38671 } } peers { permanent_uuid: "ce2d44319f6d48ee894f1d5bfc293a78" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39419 } } peers { permanent_uuid: "54e13c6e85e747aea02cf40981ed4499" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34337 } }
I20260620 04:28:16.174770 16520 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39499
I20260620 04:28:16.174999 16520 heartbeater.cc:464] Registering TS with master...
I20260620 04:28:16.175235 16412 ts_tablet_manager.cc:1428] T bd6f7e2efef74b1b9026d3d982f0ee83 P 54e13c6e85e747aea02cf40981ed4499: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 04:28:16.175537 16520 heartbeater.cc:511] Master 127.0.98.62:39499 requested a full tablet report, sending...
I20260620 04:28:16.178007 16332 ts_manager.cc:194] Registered new tserver with Master: 54e13c6e85e747aea02cf40981ed4499 (127.0.98.1:34337)
I20260620 04:28:16.181314 16332 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:40935
I20260620 04:28:16.183701 16520 heartbeater.cc:503] Master 127.0.98.62:39499 was elected leader, sending a full tablet report...
Jun 20 04:28:16 dist-test-slave-nk6d krb5kdc[15751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929695, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 04:28:16.191195 16527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:16.180692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34337 (local address 127.0.0.1:49150)
0620 04:28:16.181439 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0620 04:28:16.181487 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 04:28:16.181630 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:28:16.182161 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:28:16.182166 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:28:16.182567 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:28:16.182696 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:28:16.182704 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:28:16.183943 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:28:16.183947 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:28:16.184878 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:28:16.184889 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:28:16.185539 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:28:16.185556 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:28:16.185663 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:28:16.186174 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:28:16.186188 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:28:16.187684 (+ 1496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:28:16.189591 (+ 1907us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:28:16.189599 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:28:16.189604 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:28:16.189821 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:28:16.190087 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:28:16.190090 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:28:16.190092 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:28:16.190170 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:28:16.190591 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:28:16.190597 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:28:16.190711 (+ 114us) client_negotiation.cc:769] Sending connection context
0620 04:28:16.190828 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 04:28:16.191017 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":123,"threads_started":1}
I20260620 04:28:16.191206 16528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:16.180806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49150 (local address 127.0.98.1:34337)
0620 04:28:16.181187 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0620 04:28:16.181198 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 04:28:16.181583 (+ 385us) server_negotiation.cc:407] Connection header received
0620 04:28:16.181779 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:28:16.181784 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:28:16.181981 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:28:16.182112 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:28:16.182828 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:28:16.183836 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:28:16.185103 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:28:16.185448 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:28:16.185528 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:28:16.187836 (+ 2308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:28:16.187845 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:28:16.187854 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:28:16.187875 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:28:16.189490 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:28:16.189922 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:28:16.189928 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:28:16.189930 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:28:16.190007 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:28:16.190245 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:28:16.190248 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:28:16.190250 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:28:16.190485 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:28:16.190599 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 04:28:16.190918 (+ 319us) server_negotiation.cc:299] Negotiation successful
0620 04:28:16.191065 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":105,"threads_started":1}
I20260620 04:28:16.192360 16454 tablet_service.cc:1514] Processing DeleteTablet for tablet bd6f7e2efef74b1b9026d3d982f0ee83 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:27:55 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49150
I20260620 04:28:16.193306 16529 tablet_replica.cc:330] stopping tablet replica
I20260620 04:28:16.193559 16529 raft_consensus.cc:2229] T bd6f7e2efef74b1b9026d3d982f0ee83 P 54e13c6e85e747aea02cf40981ed4499 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:28:16.193781 16529 raft_consensus.cc:2258] T bd6f7e2efef74b1b9026d3d982f0ee83 P 54e13c6e85e747aea02cf40981ed4499 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:28:16.194626 16529 ts_tablet_manager.cc:1905] T bd6f7e2efef74b1b9026d3d982f0ee83 P 54e13c6e85e747aea02cf40981ed4499: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:28:16.198016 16529 ts_tablet_manager.cc:1918] T bd6f7e2efef74b1b9026d3d982f0ee83 P 54e13c6e85e747aea02cf40981ed4499: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:28:16.198102 16529 log.cc:1196] T bd6f7e2efef74b1b9026d3d982f0ee83 P 54e13c6e85e747aea02cf40981ed4499: Deleting WAL directory at /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-0/wal/wals/bd6f7e2efef74b1b9026d3d982f0ee83
I20260620 04:28:16.198323 16529 ts_tablet_manager.cc:1939] T bd6f7e2efef74b1b9026d3d982f0ee83 P 54e13c6e85e747aea02cf40981ed4499: Deleting consensus metadata
I20260620 04:28:16.199833 16318 catalog_manager.cc:4841] TS 54e13c6e85e747aea02cf40981ed4499 (127.0.98.1:34337): tablet bd6f7e2efef74b1b9026d3d982f0ee83 (table test-table [id=91aaa862b6a648a885dedeabfa907fb1]) successfully deleted
W20260620 04:28:16.266803 16523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:28:16.267045 16523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:28:16.267086 16523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:28:16.271075 16523 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:28:16.271133 16523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:28:16.271191 16523 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 04:28:16.276412 16523 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:38671
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=43673
--tserver_master_addrs=127.0.98.62:39499
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16523
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:28:16.277436 16523 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:28:16.278434 16523 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:28:16.287810 16535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:28:16.287806 16534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:28:16.287798 16537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:28:16.288312 16523 server_base.cc:1034] running on GCE node
I20260620 04:28:16.288729 16523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:28:16.289431 16523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:28:16.290601 16523 hybrid_clock.cc:648] HybridClock initialized: now 1781929696290571 us; error 46 us; skew 500 ppm
Jun 20 04:28:16 dist-test-slave-nk6d krb5kdc[15751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929696, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:28:16.293265 16523 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 04:28:16.294097 16523 webserver.cc:460] Webserver started at http://127.0.98.2:43673/ using document root <none> and password file <none>
I20260620 04:28:16.294625 16523 fs_manager.cc:359] Metadata directory not provided
I20260620 04:28:16.294698 16523 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:28:16.298556 16523 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 04:28:16.300730 16544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:28:16.301918 16523 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:28:16.302014 16523 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-1/wal
uuid: "a546aa48667e483bbbde10dfcdf180ff"
format_stamp: "Formatted at 2026-06-20 04:27:54 on dist-test-slave-nk6d"
server_key: "417962fc6e424d7083ac63c3da11ea99"
server_key_iv: "4c9b56092034d7f31b4d4353ba22ed2b"
server_key_version: "encryptionkey@0"
I20260620 04:28:16.302546 16523 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:28:16.314885 16523 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:28:16.317890 16523 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:28:16.318051 16523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:28:16.318602 16523 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:28:16.319597 16551 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 04:28:16.323982 16523 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 04:28:16.324047 16523 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260620 04:28:16.324101 16523 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 04:28:16.325729 16523 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 04:28:16.325775 16523 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260620 04:28:16.326035 16551 tablet_bootstrap.cc:492] T bd6f7e2efef74b1b9026d3d982f0ee83 P a546aa48667e483bbbde10dfcdf180ff: Bootstrap starting.
I20260620 04:28:16.331774 16551 log.cc:824] T bd6f7e2efef74b1b9026d3d982f0ee83 P a546aa48667e483bbbde10dfcdf180ff: Log is configured to *not* fsync() on all Append() calls
I20260620 04:28:16.335163 16523 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38671
I20260620 04:28:16.335222 16659 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38671 every 8 connection(s)
I20260620 04:28:16.335992 16523 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-1/data/info.pb
I20260620 04:28:16.339687 16551 tablet_bootstrap.cc:492] T bd6f7e2efef74b1b9026d3d982f0ee83 P a546aa48667e483bbbde10dfcdf180ff: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 04:28:16.339700 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 16523
I20260620 04:28:16.340107 16551 tablet_bootstrap.cc:492] T bd6f7e2efef74b1b9026d3d982f0ee83 P a546aa48667e483bbbde10dfcdf180ff: Bootstrap complete.
I20260620 04:28:16.340179 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:39419
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:39499
--webserver_port=44699
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 04:28:16.340461 16551 ts_tablet_manager.cc:1397] T bd6f7e2efef74b1b9026d3d982f0ee83 P a546aa48667e483bbbde10dfcdf180ff: Time spent bootstrapping tablet: real 0.015s user 0.010s sys 0.004s
I20260620 04:28:16.343130 16551 raft_consensus.cc:348] T bd6f7e2efef74b1b9026d3d982f0ee83 P a546aa48667e483bbbde10dfcdf180ff [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a546aa48667e483bbbde10dfcdf180ff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38671 } } peers { permanent_uuid: "ce2d44319f6d48ee894f1d5bfc293a78" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39419 } } peers { permanent_uuid: "54e13c6e85e747aea02cf40981ed4499" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34337 } }
I20260620 04:28:16.343612 16551 raft_consensus.cc:729] T bd6f7e2efef74b1b9026d3d982f0ee83 P a546aa48667e483bbbde10dfcdf180ff [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a546aa48667e483bbbde10dfcdf180ff, State: Initialized, Role: FOLLOWER
I20260620 04:28:16.344146 16551 consensus_queue.cc:260] T bd6f7e2efef74b1b9026d3d982f0ee83 P a546aa48667e483bbbde10dfcdf180ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a546aa48667e483bbbde10dfcdf180ff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38671 } } peers { permanent_uuid: "ce2d44319f6d48ee894f1d5bfc293a78" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39419 } } peers { permanent_uuid: "54e13c6e85e747aea02cf40981ed4499" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34337 } }
I20260620 04:28:16.345240 16551 ts_tablet_manager.cc:1428] T bd6f7e2efef74b1b9026d3d982f0ee83 P a546aa48667e483bbbde10dfcdf180ff: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
Jun 20 04:28:16 dist-test-slave-nk6d krb5kdc[15751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929696, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:28:16.350376 16524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:16.337808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49109 (local address 127.0.98.62:39499)
0620 04:28:16.337954 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 04:28:16.337959 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:28:16.338540 (+ 581us) server_negotiation.cc:407] Connection header received
0620 04:28:16.339501 (+ 961us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:28:16.339505 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:28:16.339574 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:28:16.339682 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:28:16.341175 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:28:16.341994 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:28:16.343282 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:28:16.343616 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:28:16.343746 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:28:16.346930 (+ 3184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:28:16.346933 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:28:16.346935 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:28:16.346964 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:28:16.348441 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:28:16.348919 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:28:16.348922 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:28:16.348923 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:28:16.349002 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:28:16.349318 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:28:16.349321 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:28:16.349323 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:28:16.349508 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:28:16.349596 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 04:28:16.350077 (+ 481us) server_negotiation.cc:299] Negotiation successful
0620 04:28:16.350210 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 04:28:16.351140 16662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:16.337924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39499 (local address 127.0.98.2:49109)
0620 04:28:16.338381 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0620 04:28:16.338432 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 04:28:16.339274 (+ 842us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:28:16.339860 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:28:16.339874 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:28:16.340354 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:28:16.341002 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:28:16.341020 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:28:16.342174 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:28:16.342179 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:28:16.343159 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:28:16.343170 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:28:16.343778 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:28:16.343782 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:28:16.343997 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:28:16.345112 (+ 1115us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:28:16.345132 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:28:16.346755 (+ 1623us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:28:16.348551 (+ 1796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:28:16.348556 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:28:16.348572 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:28:16.348820 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:28:16.349085 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:28:16.349089 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:28:16.349091 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:28:16.349233 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:28:16.349608 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:28:16.349615 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:28:16.349859 (+ 244us) client_negotiation.cc:769] Sending connection context
0620 04:28:16.350021 (+ 162us) client_negotiation.cc:240] Negotiation successful
0620 04:28:16.350351 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":111,"threads_started":1}
I20260620 04:28:16.352402 16660 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39499
I20260620 04:28:16.352617 16660 heartbeater.cc:464] Registering TS with master...
I20260620 04:28:16.353147 16660 heartbeater.cc:511] Master 127.0.98.62:39499 requested a full tablet report, sending...
I20260620 04:28:16.354837 16332 ts_manager.cc:194] Registered new tserver with Master: a546aa48667e483bbbde10dfcdf180ff (127.0.98.2:38671)
I20260620 04:28:16.356082 16332 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:49109
I20260620 04:28:16.358312 16660 heartbeater.cc:503] Master 127.0.98.62:39499 was elected leader, sending a full tablet report...
Jun 20 04:28:16 dist-test-slave-nk6d krb5kdc[15751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929695, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260620 04:28:16.364816 16527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:16.355702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38671 (local address 127.0.0.1:53980)
0620 04:28:16.356382 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0620 04:28:16.356395 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:28:16.356474 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:28:16.356874 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:28:16.356877 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:28:16.357054 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:28:16.357135 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:28:16.357140 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:28:16.358152 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:28:16.358155 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:28:16.358777 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:28:16.358787 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:28:16.359894 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:28:16.359909 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:28:16.360033 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:28:16.360429 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:28:16.360442 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:28:16.361534 (+ 1092us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:28:16.363352 (+ 1818us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:28:16.363355 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:28:16.363357 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:28:16.363541 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:28:16.363822 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:28:16.363825 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:28:16.363827 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:28:16.363905 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:28:16.364335 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:28:16.364338 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:28:16.364432 (+ 94us) client_negotiation.cc:769] Sending connection context
0620 04:28:16.364518 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 04:28:16.364654 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613}
I20260620 04:28:16.364909 16665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:16.355838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53980 (local address 127.0.98.2:38671)
0620 04:28:16.356127 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0620 04:28:16.356136 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 04:28:16.356455 (+ 319us) server_negotiation.cc:407] Connection header received
0620 04:28:16.356567 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:28:16.356572 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:28:16.356756 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:28:16.356873 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:28:16.357256 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:28:16.358048 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:28:16.359401 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:28:16.359801 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:28:16.359894 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:28:16.361657 (+ 1763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:28:16.361665 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:28:16.361755 (+ 90us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:28:16.361782 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:28:16.363248 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:28:16.363648 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:28:16.363656 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:28:16.363657 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:28:16.363730 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:28:16.364000 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:28:16.364004 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:28:16.364005 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:28:16.364234 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:28:16.364350 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 04:28:16.364606 (+ 256us) server_negotiation.cc:299] Negotiation successful
0620 04:28:16.364763 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":89,"threads_started":1}
I20260620 04:28:16.365751 16593 tablet_service.cc:1514] Processing DeleteTablet for tablet bd6f7e2efef74b1b9026d3d982f0ee83 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:27:55 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53980
I20260620 04:28:16.366791 16666 tablet_replica.cc:330] stopping tablet replica
I20260620 04:28:16.367048 16666 raft_consensus.cc:2229] T bd6f7e2efef74b1b9026d3d982f0ee83 P a546aa48667e483bbbde10dfcdf180ff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:28:16.367267 16666 raft_consensus.cc:2258] T bd6f7e2efef74b1b9026d3d982f0ee83 P a546aa48667e483bbbde10dfcdf180ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:28:16.368196 16666 ts_tablet_manager.cc:1905] T bd6f7e2efef74b1b9026d3d982f0ee83 P a546aa48667e483bbbde10dfcdf180ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:28:16.371568 16666 ts_tablet_manager.cc:1918] T bd6f7e2efef74b1b9026d3d982f0ee83 P a546aa48667e483bbbde10dfcdf180ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:28:16.371651 16666 log.cc:1196] T bd6f7e2efef74b1b9026d3d982f0ee83 P a546aa48667e483bbbde10dfcdf180ff: Deleting WAL directory at /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-1/wal/wals/bd6f7e2efef74b1b9026d3d982f0ee83
I20260620 04:28:16.371884 16666 ts_tablet_manager.cc:1939] T bd6f7e2efef74b1b9026d3d982f0ee83 P a546aa48667e483bbbde10dfcdf180ff: Deleting consensus metadata
I20260620 04:28:16.373420 16316 catalog_manager.cc:4841] TS a546aa48667e483bbbde10dfcdf180ff (127.0.98.2:38671): tablet bd6f7e2efef74b1b9026d3d982f0ee83 (table test-table [id=91aaa862b6a648a885dedeabfa907fb1]) successfully deleted
W20260620 04:28:16.452706 16663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:28:16.452955 16663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:28:16.452998 16663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:28:16.456923 16663 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:28:16.456979 16663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:28:16.457041 16663 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 04:28:16.462231 16663 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:39419
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=44699
--tserver_master_addrs=127.0.98.62:39499
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16663
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:28:16.463304 16663 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:28:16.464357 16663 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:28:16.473707 16674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:28:16.473824 16671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:28:16.473747 16672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:28:16.473934 16663 server_base.cc:1034] running on GCE node
I20260620 04:28:16.474387 16663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:28:16.475013 16663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:28:16.476150 16663 hybrid_clock.cc:648] HybridClock initialized: now 1781929696476123 us; error 42 us; skew 500 ppm
Jun 20 04:28:16 dist-test-slave-nk6d krb5kdc[15751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929696, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:28:16.478869 16663 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 04:28:16.479683 16663 webserver.cc:460] Webserver started at http://127.0.98.3:44699/ using document root <none> and password file <none>
I20260620 04:28:16.480265 16663 fs_manager.cc:359] Metadata directory not provided
I20260620 04:28:16.480356 16663 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:28:16.484213 16663 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 04:28:16.486199 16681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:28:16.487470 16663 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:28:16.487561 16663 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-2/wal
uuid: "ce2d44319f6d48ee894f1d5bfc293a78"
format_stamp: "Formatted at 2026-06-20 04:27:55 on dist-test-slave-nk6d"
server_key: "325daa7963f8f69e62f6445288a6a072"
server_key_iv: "815942cb08a3453b062f0e0623e31ad6"
server_key_version: "encryptionkey@0"
I20260620 04:28:16.488150 16663 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:28:16.520380 16663 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:28:16.523170 16663 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:28:16.523342 16663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:28:16.523914 16663 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:28:16.524731 16663 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:28:16.524775 16663 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:28:16.524827 16663 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:28:16.524856 16663 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:28:16.533056 16663 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39419
I20260620 04:28:16.533089 16794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39419 every 8 connection(s)
I20260620 04:28:16.534034 16663 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-2/data/info.pb
I20260620 04:28:16.538133 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 16663
Jun 20 04:28:16 dist-test-slave-nk6d krb5kdc[15751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929696, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:28:16.548627 16524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:16.535807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57853 (local address 127.0.98.62:39499)
0620 04:28:16.535982 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 04:28:16.535987 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:28:16.536844 (+ 857us) server_negotiation.cc:407] Connection header received
0620 04:28:16.538075 (+ 1231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:28:16.538078 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:28:16.538136 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:28:16.538209 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:28:16.539537 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:28:16.540099 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:28:16.541285 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:28:16.541601 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:28:16.541687 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:28:16.544569 (+ 2882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:28:16.544573 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:28:16.544576 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:28:16.544616 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:28:16.546310 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:28:16.546899 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:28:16.546903 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:28:16.546905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:28:16.546988 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:28:16.547344 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:28:16.547363 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:28:16.547365 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:28:16.547618 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:28:16.547719 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 04:28:16.548333 (+ 614us) server_negotiation.cc:299] Negotiation successful
0620 04:28:16.548462 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 04:28:16.549567 16797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:16.536280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39499 (local address 127.0.98.3:57853)
0620 04:28:16.536700 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0620 04:28:16.536743 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 04:28:16.537863 (+ 1120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:28:16.538397 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:28:16.538404 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:28:16.538889 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:28:16.539393 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:28:16.539407 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:28:16.540209 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:28:16.540212 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:28:16.541166 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:28:16.541177 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:28:16.541845 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:28:16.541849 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:28:16.542056 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:28:16.542612 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:28:16.542634 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:28:16.544390 (+ 1756us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:28:16.546430 (+ 2040us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:28:16.546436 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:28:16.546454 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:28:16.546788 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:28:16.547074 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:28:16.547077 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:28:16.547079 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:28:16.547253 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:28:16.547715 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:28:16.547736 (+ 21us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:28:16.548066 (+ 330us) client_negotiation.cc:769] Sending connection context
0620 04:28:16.548265 (+ 199us) client_negotiation.cc:240] Negotiation successful
0620 04:28:16.548576 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":101,"threads_started":1}
I20260620 04:28:16.551311 16795 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39499
I20260620 04:28:16.551589 16795 heartbeater.cc:464] Registering TS with master...
I20260620 04:28:16.551911 16799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:16.538781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36882 (local address 127.0.98.62:39499)
0620 04:28:16.539014 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0620 04:28:16.539018 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:28:16.539107 (+ 89us) server_negotiation.cc:407] Connection header received
0620 04:28:16.539245 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:28:16.539247 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:28:16.539286 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:28:16.539353 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:28:16.540097 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:28:16.540606 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:28:16.541632 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:28:16.541912 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:28:16.542010 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:28:16.543585 (+ 1575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:28:16.543616 (+ 31us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:28:16.543620 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:28:16.543650 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:28:16.549631 (+ 5981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:28:16.550078 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:28:16.550082 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:28:16.550083 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:28:16.550172 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:28:16.551024 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:28:16.551028 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:28:16.551029 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:28:16.551258 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:28:16.551451 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0620 04:28:16.551639 (+ 188us) server_negotiation.cc:299] Negotiation successful
0620 04:28:16.551764 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":76,"threads_started":1}
I20260620 04:28:16.552253 16795 heartbeater.cc:511] Master 127.0.98.62:39499 requested a full tablet report, sending...
I20260620 04:28:16.553354 16328 ts_manager.cc:194] Registered new tserver with Master: ce2d44319f6d48ee894f1d5bfc293a78 (127.0.98.3:39419)
I20260620 04:28:16.554023 16328 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57853
I20260620 04:28:16.565532 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 04:28:16.569481 16799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:16.566110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36898 (local address 127.0.98.62:39499)
0620 04:28:16.566235 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 04:28:16.566238 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:28:16.566389 (+ 151us) server_negotiation.cc:407] Connection header received
0620 04:28:16.566521 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:28:16.566524 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:28:16.566621 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:28:16.566705 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:28:16.567017 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:28:16.567529 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:28:16.568438 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:28:16.568665 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:28:16.568744 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:28:16.569014 (+ 270us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:28:16.569152 (+ 138us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0620 04:28:16.569357 (+ 205us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36898: token expired
Metrics: {"server-negotiator.queue_time_us":46}
W20260620 04:28:16.569543 16799 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36898: token expired
W20260620 04:28:16.569866 16260 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.98.62:39499)
I20260620 04:28:16.576362 16799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:16.570180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36900 (local address 127.0.98.62:39499)
0620 04:28:16.570282 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0620 04:28:16.570286 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:28:16.570297 (+ 11us) server_negotiation.cc:407] Connection header received
0620 04:28:16.570381 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:28:16.570383 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:28:16.570417 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:28:16.570475 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:28:16.571006 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:28:16.571467 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:28:16.572304 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:28:16.572520 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:28:16.572596 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:28:16.573731 (+ 1135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:28:16.573735 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:28:16.573737 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:28:16.573760 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:28:16.574976 (+ 1216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:28:16.575328 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:28:16.575331 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:28:16.575332 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:28:16.575405 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:28:16.575621 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:28:16.575625 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:28:16.575626 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:28:16.575803 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:28:16.575907 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 04:28:16.576143 (+ 236us) server_negotiation.cc:299] Negotiation successful
0620 04:28:16.576239 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 04:28:16.578759 16328 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36900:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 04:28:16.582031 16328 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 04:28:16.588953 16593 tablet_service.cc:1467] Processing CreateTablet for tablet df93fbb39b82405dbf56853d5de3b70d (DEFAULT_TABLE table=test-table [id=dcabc00fdbc94fb0862dd8ae174459e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:28:16.589056 16454 tablet_service.cc:1467] Processing CreateTablet for tablet df93fbb39b82405dbf56853d5de3b70d (DEFAULT_TABLE table=test-table [id=dcabc00fdbc94fb0862dd8ae174459e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:28:16.589596 16593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df93fbb39b82405dbf56853d5de3b70d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:28:16.589643 16454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df93fbb39b82405dbf56853d5de3b70d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:28:16.592208 16412 tablet_bootstrap.cc:492] T df93fbb39b82405dbf56853d5de3b70d P 54e13c6e85e747aea02cf40981ed4499: Bootstrap starting.
I20260620 04:28:16.592317 16551 tablet_bootstrap.cc:492] T df93fbb39b82405dbf56853d5de3b70d P a546aa48667e483bbbde10dfcdf180ff: Bootstrap starting.
I20260620 04:28:16.592991 16412 tablet_bootstrap.cc:654] T df93fbb39b82405dbf56853d5de3b70d P 54e13c6e85e747aea02cf40981ed4499: Neither blocks nor log segments found. Creating new log.
I20260620 04:28:16.593202 16551 tablet_bootstrap.cc:654] T df93fbb39b82405dbf56853d5de3b70d P a546aa48667e483bbbde10dfcdf180ff: Neither blocks nor log segments found. Creating new log.
I20260620 04:28:16.593961 16527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:16.588257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39419 (local address 127.0.0.1:41860)
0620 04:28:16.588422 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 04:28:16.588436 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:28:16.588517 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:28:16.589629 (+ 1112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:28:16.589632 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:28:16.589651 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:28:16.589765 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:28:16.589772 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:28:16.591221 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:28:16.591225 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:28:16.592434 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:28:16.592445 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:28:16.593559 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:28:16.593563 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:28:16.593654 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:28:16.593705 (+ 51us) client_negotiation.cc:769] Sending connection context
0620 04:28:16.593759 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 04:28:16.593835 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95}
I20260620 04:28:16.594540 16804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:16.588695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41860 (local address 127.0.98.3:39419)
0620 04:28:16.589116 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0620 04:28:16.589128 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 04:28:16.589154 (+ 26us) server_negotiation.cc:407] Connection header received
0620 04:28:16.589232 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:28:16.589239 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:28:16.589484 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:28:16.589650 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:28:16.589913 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:28:16.591025 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:28:16.592565 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:28:16.593417 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:28:16.594122 (+ 705us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:28:16.594285 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 04:28:16.594328 (+ 43us) server_negotiation.cc:299] Negotiation successful
0620 04:28:16.594399 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":196,"threads_started":1}
I20260620 04:28:16.594832 16412 tablet_bootstrap.cc:492] T df93fbb39b82405dbf56853d5de3b70d P 54e13c6e85e747aea02cf40981ed4499: No bootstrap required, opened a new log
I20260620 04:28:16.594929 16412 ts_tablet_manager.cc:1397] T df93fbb39b82405dbf56853d5de3b70d P 54e13c6e85e747aea02cf40981ed4499: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 04:28:16.594949 16551 tablet_bootstrap.cc:492] T df93fbb39b82405dbf56853d5de3b70d P a546aa48667e483bbbde10dfcdf180ff: No bootstrap required, opened a new log
I20260620 04:28:16.595018 16551 ts_tablet_manager.cc:1397] T df93fbb39b82405dbf56853d5de3b70d P a546aa48667e483bbbde10dfcdf180ff: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 04:28:16.595360 16412 raft_consensus.cc:348] T df93fbb39b82405dbf56853d5de3b70d P 54e13c6e85e747aea02cf40981ed4499 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce2d44319f6d48ee894f1d5bfc293a78" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39419 } } peers { permanent_uuid: "a546aa48667e483bbbde10dfcdf180ff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38671 } } peers { permanent_uuid: "54e13c6e85e747aea02cf40981ed4499" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34337 } }
I20260620 04:28:16.595371 16551 raft_consensus.cc:348] T df93fbb39b82405dbf56853d5de3b70d P a546aa48667e483bbbde10dfcdf180ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce2d44319f6d48ee894f1d5bfc293a78" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39419 } } peers { permanent_uuid: "a546aa48667e483bbbde10dfcdf180ff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38671 } } peers { permanent_uuid: "54e13c6e85e747aea02cf40981ed4499" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34337 } }
I20260620 04:28:16.595479 16551 raft_consensus.cc:374] T df93fbb39b82405dbf56853d5de3b70d P a546aa48667e483bbbde10dfcdf180ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:28:16.595480 16412 raft_consensus.cc:374] T df93fbb39b82405dbf56853d5de3b70d P 54e13c6e85e747aea02cf40981ed4499 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:28:16.595523 16551 raft_consensus.cc:729] T df93fbb39b82405dbf56853d5de3b70d P a546aa48667e483bbbde10dfcdf180ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a546aa48667e483bbbde10dfcdf180ff, State: Initialized, Role: FOLLOWER
I20260620 04:28:16.595523 16412 raft_consensus.cc:729] T df93fbb39b82405dbf56853d5de3b70d P 54e13c6e85e747aea02cf40981ed4499 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54e13c6e85e747aea02cf40981ed4499, State: Initialized, Role: FOLLOWER
I20260620 04:28:16.595649 16551 consensus_queue.cc:260] T df93fbb39b82405dbf56853d5de3b70d P a546aa48667e483bbbde10dfcdf180ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce2d44319f6d48ee894f1d5bfc293a78" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39419 } } peers { permanent_uuid: "a546aa48667e483bbbde10dfcdf180ff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38671 } } peers { permanent_uuid: "54e13c6e85e747aea02cf40981ed4499" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34337 } }
I20260620 04:28:16.595646 16412 consensus_queue.cc:260] T df93fbb39b82405dbf56853d5de3b70d P 54e13c6e85e747aea02cf40981ed4499 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce2d44319f6d48ee894f1d5bfc293a78" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39419 } } peers { permanent_uuid: "a546aa48667e483bbbde10dfcdf180ff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38671 } } peers { permanent_uuid: "54e13c6e85e747aea02cf40981ed4499" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34337 } }
I20260620 04:28:16.595890 16551 ts_tablet_manager.cc:1428] T df93fbb39b82405dbf56853d5de3b70d P a546aa48667e483bbbde10dfcdf180ff: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 04:28:16.595942 16412 ts_tablet_manager.cc:1428] T df93fbb39b82405dbf56853d5de3b70d P 54e13c6e85e747aea02cf40981ed4499: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 04:28:16.597769 16729 tablet_service.cc:1467] Processing CreateTablet for tablet df93fbb39b82405dbf56853d5de3b70d (DEFAULT_TABLE table=test-table [id=dcabc00fdbc94fb0862dd8ae174459e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:28:16.598759 16729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df93fbb39b82405dbf56853d5de3b70d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:28:16.605175 16807 tablet_bootstrap.cc:492] T df93fbb39b82405dbf56853d5de3b70d P ce2d44319f6d48ee894f1d5bfc293a78: Bootstrap starting.
I20260620 04:28:16.607257 16807 tablet_bootstrap.cc:654] T df93fbb39b82405dbf56853d5de3b70d P ce2d44319f6d48ee894f1d5bfc293a78: Neither blocks nor log segments found. Creating new log.
I20260620 04:28:16.608289 16807 log.cc:824] T df93fbb39b82405dbf56853d5de3b70d P ce2d44319f6d48ee894f1d5bfc293a78: Log is configured to *not* fsync() on all Append() calls
I20260620 04:28:16.610139 16807 tablet_bootstrap.cc:492] T df93fbb39b82405dbf56853d5de3b70d P ce2d44319f6d48ee894f1d5bfc293a78: No bootstrap required, opened a new log
I20260620 04:28:16.610363 16807 ts_tablet_manager.cc:1397] T df93fbb39b82405dbf56853d5de3b70d P ce2d44319f6d48ee894f1d5bfc293a78: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 04:28:16.614451 16807 raft_consensus.cc:348] T df93fbb39b82405dbf56853d5de3b70d P ce2d44319f6d48ee894f1d5bfc293a78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce2d44319f6d48ee894f1d5bfc293a78" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39419 } } peers { permanent_uuid: "a546aa48667e483bbbde10dfcdf180ff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38671 } } peers { permanent_uuid: "54e13c6e85e747aea02cf40981ed4499" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34337 } }
I20260620 04:28:16.614789 16807 raft_consensus.cc:374] T df93fbb39b82405dbf56853d5de3b70d P ce2d44319f6d48ee894f1d5bfc293a78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:28:16.614846 16807 raft_consensus.cc:729] T df93fbb39b82405dbf56853d5de3b70d P ce2d44319f6d48ee894f1d5bfc293a78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce2d44319f6d48ee894f1d5bfc293a78, State: Initialized, Role: FOLLOWER
I20260620 04:28:16.615310 16807 consensus_queue.cc:260] T df93fbb39b82405dbf56853d5de3b70d P ce2d44319f6d48ee894f1d5bfc293a78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce2d44319f6d48ee894f1d5bfc293a78" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39419 } } peers { permanent_uuid: "a546aa48667e483bbbde10dfcdf180ff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38671 } } peers { permanent_uuid: "54e13c6e85e747aea02cf40981ed4499" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34337 } }
I20260620 04:28:16.616062 16795 heartbeater.cc:503] Master 127.0.98.62:39499 was elected leader, sending a full tablet report...
I20260620 04:28:16.616253 16807 ts_tablet_manager.cc:1428] T df93fbb39b82405dbf56853d5de3b70d P ce2d44319f6d48ee894f1d5bfc293a78: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
W20260620 04:28:16.652601 16521 tablet.cc:2369] T df93fbb39b82405dbf56853d5de3b70d P 54e13c6e85e747aea02cf40981ed4499: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 04:28:16.774885 16526 raft_consensus.cc:482] T df93fbb39b82405dbf56853d5de3b70d P 54e13c6e85e747aea02cf40981ed4499 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:28:16.775161 16526 raft_consensus.cc:504] T df93fbb39b82405dbf56853d5de3b70d P 54e13c6e85e747aea02cf40981ed4499 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce2d44319f6d48ee894f1d5bfc293a78" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39419 } } peers { permanent_uuid: "a546aa48667e483bbbde10dfcdf180ff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38671 } } peers { permanent_uuid: "54e13c6e85e747aea02cf40981ed4499" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34337 } }
I20260620 04:28:16.776484 16526 leader_election.cc:288] T df93fbb39b82405dbf56853d5de3b70d P 54e13c6e85e747aea02cf40981ed4499 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ce2d44319f6d48ee894f1d5bfc293a78 (127.0.98.3:39419), a546aa48667e483bbbde10dfcdf180ff (127.0.98.2:38671)
I20260620 04:28:16.781399 16804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:16.776852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44101 (local address 127.0.98.3:39419)
0620 04:28:16.776994 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 04:28:16.776998 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:28:16.777121 (+ 123us) server_negotiation.cc:407] Connection header received
0620 04:28:16.777308 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:28:16.777310 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:28:16.777359 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:28:16.777440 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:28:16.777759 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:28:16.778573 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:28:16.780187 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:28:16.780794 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:28:16.780933 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:28:16.780996 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 04:28:16.781207 (+ 211us) server_negotiation.cc:299] Negotiation successful
0620 04:28:16.781271 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 04:28:16.781436 16665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:16.776826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44279 (local address 127.0.98.2:38671)
0620 04:28:16.776950 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 04:28:16.776955 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:28:16.777122 (+ 167us) server_negotiation.cc:407] Connection header received
0620 04:28:16.777307 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:28:16.777310 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:28:16.777372 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:28:16.777468 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:28:16.777759 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:28:16.778569 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:28:16.780187 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:28:16.780836 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:28:16.780927 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:28:16.781080 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 04:28:16.781201 (+ 121us) server_negotiation.cc:299] Negotiation successful
0620 04:28:16.781271 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 04:28:16.781476 16810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:16.776682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39419 (local address 127.0.98.1:44101)
0620 04:28:16.777033 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0620 04:28:16.777047 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:28:16.777183 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:28:16.777455 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:28:16.777458 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:28:16.777478 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:28:16.777658 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:28:16.777665 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:28:16.778675 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:28:16.778678 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:28:16.780076 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:28:16.780082 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:28:16.780926 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:28:16.780930 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:28:16.781052 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:28:16.781121 (+ 69us) client_negotiation.cc:769] Sending connection context
0620 04:28:16.781188 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 04:28:16.781291 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"mutex_wait_us":205,"spinlock_wait_cycles":23168,"thread_start_us":68,"threads_started":1}
I20260620 04:28:16.781481 16811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:16.776735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38671 (local address 127.0.98.1:44279)
0620 04:28:16.777033 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0620 04:28:16.777047 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:28:16.777183 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:28:16.777478 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:28:16.777480 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:28:16.777493 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:28:16.777661 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:28:16.777666 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:28:16.778676 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:28:16.778679 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:28:16.780069 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:28:16.780080 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:28:16.780931 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:28:16.780934 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:28:16.781048 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:28:16.781115 (+ 67us) client_negotiation.cc:769] Sending connection context
0620 04:28:16.781188 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 04:28:16.781291 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"mutex_wait_us":110,"spinlock_wait_cycles":9856,"thread_start_us":91,"threads_started":1}
I20260620 04:28:16.782122 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df93fbb39b82405dbf56853d5de3b70d" candidate_uuid: "54e13c6e85e747aea02cf40981ed4499" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce2d44319f6d48ee894f1d5bfc293a78" is_pre_election: true
I20260620 04:28:16.782120 16613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df93fbb39b82405dbf56853d5de3b70d" candidate_uuid: "54e13c6e85e747aea02cf40981ed4499" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a546aa48667e483bbbde10dfcdf180ff" is_pre_election: true
I20260620 04:28:16.782421 16749 raft_consensus.cc:2454] T df93fbb39b82405dbf56853d5de3b70d P ce2d44319f6d48ee894f1d5bfc293a78 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54e13c6e85e747aea02cf40981ed4499 in term 0.
I20260620 04:28:16.782423 16613 raft_consensus.cc:2454] T df93fbb39b82405dbf56853d5de3b70d P a546aa48667e483bbbde10dfcdf180ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54e13c6e85e747aea02cf40981ed4499 in term 0.
I20260620 04:28:16.782871 16406 leader_election.cc:302] T df93fbb39b82405dbf56853d5de3b70d P 54e13c6e85e747aea02cf40981ed4499 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54e13c6e85e747aea02cf40981ed4499, a546aa48667e483bbbde10dfcdf180ff; no voters:
I20260620 04:28:16.783149 16526 raft_consensus.cc:2781] T df93fbb39b82405dbf56853d5de3b70d P 54e13c6e85e747aea02cf40981ed4499 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 04:28:16.783229 16526 raft_consensus.cc:482] T df93fbb39b82405dbf56853d5de3b70d P 54e13c6e85e747aea02cf40981ed4499 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 04:28:16.783259 16526 raft_consensus.cc:3037] T df93fbb39b82405dbf56853d5de3b70d P 54e13c6e85e747aea02cf40981ed4499 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:28:16.784308 16526 raft_consensus.cc:504] T df93fbb39b82405dbf56853d5de3b70d P 54e13c6e85e747aea02cf40981ed4499 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce2d44319f6d48ee894f1d5bfc293a78" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39419 } } peers { permanent_uuid: "a546aa48667e483bbbde10dfcdf180ff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38671 } } peers { permanent_uuid: "54e13c6e85e747aea02cf40981ed4499" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34337 } }
I20260620 04:28:16.784659 16526 leader_election.cc:288] T df93fbb39b82405dbf56853d5de3b70d P 54e13c6e85e747aea02cf40981ed4499 [CANDIDATE]: Term 1 election: Requested vote from peers ce2d44319f6d48ee894f1d5bfc293a78 (127.0.98.3:39419), a546aa48667e483bbbde10dfcdf180ff (127.0.98.2:38671)
W20260620 04:28:16.784910 16796 tablet.cc:2369] T df93fbb39b82405dbf56853d5de3b70d P ce2d44319f6d48ee894f1d5bfc293a78: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 04:28:16.784970 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df93fbb39b82405dbf56853d5de3b70d" candidate_uuid: "54e13c6e85e747aea02cf40981ed4499" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce2d44319f6d48ee894f1d5bfc293a78"
I20260620 04:28:16.784976 16613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df93fbb39b82405dbf56853d5de3b70d" candidate_uuid: "54e13c6e85e747aea02cf40981ed4499" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a546aa48667e483bbbde10dfcdf180ff"
I20260620 04:28:16.785068 16749 raft_consensus.cc:3037] T df93fbb39b82405dbf56853d5de3b70d P ce2d44319f6d48ee894f1d5bfc293a78 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:28:16.785089 16613 raft_consensus.cc:3037] T df93fbb39b82405dbf56853d5de3b70d P a546aa48667e483bbbde10dfcdf180ff [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:28:16.786046 16749 raft_consensus.cc:2454] T df93fbb39b82405dbf56853d5de3b70d P ce2d44319f6d48ee894f1d5bfc293a78 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54e13c6e85e747aea02cf40981ed4499 in term 1.
I20260620 04:28:16.786098 16613 raft_consensus.cc:2454] T df93fbb39b82405dbf56853d5de3b70d P a546aa48667e483bbbde10dfcdf180ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54e13c6e85e747aea02cf40981ed4499 in term 1.
I20260620 04:28:16.786455 16406 leader_election.cc:302] T df93fbb39b82405dbf56853d5de3b70d P 54e13c6e85e747aea02cf40981ed4499 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54e13c6e85e747aea02cf40981ed4499, a546aa48667e483bbbde10dfcdf180ff; no voters:
I20260620 04:28:16.786628 16526 raft_consensus.cc:2781] T df93fbb39b82405dbf56853d5de3b70d P 54e13c6e85e747aea02cf40981ed4499 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:28:16.786815 16526 raft_consensus.cc:686] T df93fbb39b82405dbf56853d5de3b70d P 54e13c6e85e747aea02cf40981ed4499 [term 1 LEADER]: Becoming Leader. State: Replica: 54e13c6e85e747aea02cf40981ed4499, State: Running, Role: LEADER
I20260620 04:28:16.787087 16526 consensus_queue.cc:237] T df93fbb39b82405dbf56853d5de3b70d P 54e13c6e85e747aea02cf40981ed4499 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce2d44319f6d48ee894f1d5bfc293a78" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39419 } } peers { permanent_uuid: "a546aa48667e483bbbde10dfcdf180ff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38671 } } peers { permanent_uuid: "54e13c6e85e747aea02cf40981ed4499" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34337 } }
I20260620 04:28:16.790122 16328 catalog_manager.cc:5495] T df93fbb39b82405dbf56853d5de3b70d P 54e13c6e85e747aea02cf40981ed4499 reported cstate change: term changed from 0 to 1, leader changed from <none> to 54e13c6e85e747aea02cf40981ed4499 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "54e13c6e85e747aea02cf40981ed4499" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce2d44319f6d48ee894f1d5bfc293a78" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39419 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a546aa48667e483bbbde10dfcdf180ff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54e13c6e85e747aea02cf40981ed4499" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34337 } health_report { overall_health: HEALTHY } } }
I20260620 04:28:16.829107 16815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:16.825642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49164 (local address 127.0.98.1:34337)
0620 04:28:16.825934 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0620 04:28:16.825941 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 04:28:16.825952 (+ 11us) server_negotiation.cc:407] Connection header received
0620 04:28:16.826025 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:28:16.826028 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:28:16.826082 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:28:16.826188 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:28:16.826424 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:28:16.826929 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:28:16.827735 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:28:16.827976 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:28:16.828071 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:28:16.828353 (+ 282us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:28:16.828775 (+ 422us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:28:16.828859 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 04:28:16.828937 (+ 78us) server_negotiation.cc:299] Negotiation successful
0620 04:28:16.828983 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":84,"threads_started":1}
I20260620 04:28:16.832721 16613 raft_consensus.cc:1261] T df93fbb39b82405dbf56853d5de3b70d P a546aa48667e483bbbde10dfcdf180ff [term 1 FOLLOWER]: Refusing update from remote peer 54e13c6e85e747aea02cf40981ed4499: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:28:16.832726 16749 raft_consensus.cc:1261] T df93fbb39b82405dbf56853d5de3b70d P ce2d44319f6d48ee894f1d5bfc293a78 [term 1 FOLLOWER]: Refusing update from remote peer 54e13c6e85e747aea02cf40981ed4499: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:28:16.833328 16526 consensus_queue.cc:1035] T df93fbb39b82405dbf56853d5de3b70d P 54e13c6e85e747aea02cf40981ed4499 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce2d44319f6d48ee894f1d5bfc293a78" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39419 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:28:16.833513 16812 consensus_queue.cc:1035] T df93fbb39b82405dbf56853d5de3b70d P 54e13c6e85e747aea02cf40981ed4499 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a546aa48667e483bbbde10dfcdf180ff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 04:28:16.837083 16661 tablet.cc:2369] T df93fbb39b82405dbf56853d5de3b70d P a546aa48667e483bbbde10dfcdf180ff: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 04:28:16.845842 16328 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36900:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:28:16.846021 16328 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36900:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:28:16.848361 16328 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cccb8275856c44bbb18eacd620970116: Sending DeleteTablet for 3 replicas of tablet df93fbb39b82405dbf56853d5de3b70d
I20260620 04:28:16.848961 16593 tablet_service.cc:1514] Processing DeleteTablet for tablet df93fbb39b82405dbf56853d5de3b70d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:28:16 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53980
I20260620 04:28:16.848980 16729 tablet_service.cc:1514] Processing DeleteTablet for tablet df93fbb39b82405dbf56853d5de3b70d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:28:16 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41860
I20260620 04:28:16.849136 16666 tablet_replica.cc:330] T df93fbb39b82405dbf56853d5de3b70d P a546aa48667e483bbbde10dfcdf180ff: stopping tablet replica
I20260620 04:28:16.849125 16454 tablet_service.cc:1514] Processing DeleteTablet for tablet df93fbb39b82405dbf56853d5de3b70d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:28:16 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49150
I20260620 04:28:16.849241 16666 raft_consensus.cc:2229] T df93fbb39b82405dbf56853d5de3b70d P a546aa48667e483bbbde10dfcdf180ff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:28:16.849357 16666 raft_consensus.cc:2258] T df93fbb39b82405dbf56853d5de3b70d P a546aa48667e483bbbde10dfcdf180ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:28:16.849550 16826 tablet_replica.cc:330] T df93fbb39b82405dbf56853d5de3b70d P 54e13c6e85e747aea02cf40981ed4499: stopping tablet replica
I20260620 04:28:16.849653 16826 raft_consensus.cc:2229] T df93fbb39b82405dbf56853d5de3b70d P 54e13c6e85e747aea02cf40981ed4499 [term 1 LEADER]: Raft consensus shutting down.
I20260620 04:28:16.849684 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 16370
I20260620 04:28:16.849908 16826 raft_consensus.cc:2258] T df93fbb39b82405dbf56853d5de3b70d P 54e13c6e85e747aea02cf40981ed4499 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:28:16.850687 16666 ts_tablet_manager.cc:1905] T df93fbb39b82405dbf56853d5de3b70d P a546aa48667e483bbbde10dfcdf180ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:28:16.852759 16666 ts_tablet_manager.cc:1918] T df93fbb39b82405dbf56853d5de3b70d P a546aa48667e483bbbde10dfcdf180ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:28:16.852841 16666 log.cc:1196] T df93fbb39b82405dbf56853d5de3b70d P a546aa48667e483bbbde10dfcdf180ff: Deleting WAL directory at /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-1/wal/wals/df93fbb39b82405dbf56853d5de3b70d
I20260620 04:28:16.853082 16666 ts_tablet_manager.cc:1939] T df93fbb39b82405dbf56853d5de3b70d P a546aa48667e483bbbde10dfcdf180ff: Deleting consensus metadata
I20260620 04:28:16.853693 16316 catalog_manager.cc:4841] TS a546aa48667e483bbbde10dfcdf180ff (127.0.98.2:38671): tablet df93fbb39b82405dbf56853d5de3b70d (table test-table [id=dcabc00fdbc94fb0862dd8ae174459e1]) successfully deleted
I20260620 04:28:16.853725 16825 tablet_replica.cc:330] T df93fbb39b82405dbf56853d5de3b70d P ce2d44319f6d48ee894f1d5bfc293a78: stopping tablet replica
I20260620 04:28:16.854036 16825 raft_consensus.cc:2229] T df93fbb39b82405dbf56853d5de3b70d P ce2d44319f6d48ee894f1d5bfc293a78 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:28:16.854348 16825 raft_consensus.cc:2258] T df93fbb39b82405dbf56853d5de3b70d P ce2d44319f6d48ee894f1d5bfc293a78 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:28:16.855705 16825 ts_tablet_manager.cc:1905] T df93fbb39b82405dbf56853d5de3b70d P ce2d44319f6d48ee894f1d5bfc293a78: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 04:28:16.857293 16318 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34337 (error 108)
I20260620 04:28:16.857484 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 16523
I20260620 04:28:16.858078 16825 ts_tablet_manager.cc:1918] T df93fbb39b82405dbf56853d5de3b70d P ce2d44319f6d48ee894f1d5bfc293a78: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:28:16.858152 16825 log.cc:1196] T df93fbb39b82405dbf56853d5de3b70d P ce2d44319f6d48ee894f1d5bfc293a78: Deleting WAL directory at /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-2/wal/wals/df93fbb39b82405dbf56853d5de3b70d
I20260620 04:28:16.858412 16825 ts_tablet_manager.cc:1939] T df93fbb39b82405dbf56853d5de3b70d P ce2d44319f6d48ee894f1d5bfc293a78: Deleting consensus metadata
I20260620 04:28:16.859405 16317 catalog_manager.cc:4841] TS ce2d44319f6d48ee894f1d5bfc293a78 (127.0.98.3:39419): tablet df93fbb39b82405dbf56853d5de3b70d (table test-table [id=dcabc00fdbc94fb0862dd8ae174459e1]) successfully deleted
I20260620 04:28:16.859952 16527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:16.859708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34337 (local address 127.0.0.1:49170)
0620 04:28:16.859822 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0620 04:28:16.859884 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260620 04:28:16.860157 16318 catalog_manager.cc:4584] TS 54e13c6e85e747aea02cf40981ed4499 (127.0.98.1:34337): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet df93fbb39b82405dbf56853d5de3b70d: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34337: connect: Connection refused (error 111)
I20260620 04:28:16.865417 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 16663
I20260620 04:28:16.870216 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 16299
I20260620 04:28:36.875638 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39499
--webserver_interface=127.0.98.62
--webserver_port=38563
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39499
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:28:36.983258 16828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:28:36.983489 16828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:28:36.983525 16828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:28:36.987435 16828 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 04:28:36.987484 16828 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 04:28:36.987497 16828 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 04:28:36.987509 16828 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:28:36.987525 16828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:28:36.987543 16828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:28:36.987573 16828 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:28:36.987598 16828 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:28:36.992712 16828 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39499
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39499
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=38563
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16828
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:28:36.993686 16828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:28:36.994664 16828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:28:37.002907 16833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:28:37.002902 16836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:28:37.002905 16834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:28:37.003261 16828 server_base.cc:1034] running on GCE node
I20260620 04:28:37.003935 16828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:28:37.005054 16828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:28:37.006217 16828 hybrid_clock.cc:648] HybridClock initialized: now 1781929717006188 us; error 39 us; skew 500 ppm
Jun 20 04:28:37 dist-test-slave-nk6d krb5kdc[15751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929717, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:28:37.008934 16828 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 04:28:37.009791 16828 webserver.cc:460] Webserver started at http://127.0.98.62:38563/ using document root <none> and password file <none>
I20260620 04:28:37.010344 16828 fs_manager.cc:359] Metadata directory not provided
I20260620 04:28:37.010408 16828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:28:37.014243 16828 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260620 04:28:37.016423 16843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:28:37.017710 16828 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:28:37.017863 16828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/master-0/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/master-0/wal
uuid: "cccb8275856c44bbb18eacd620970116"
format_stamp: "Formatted at 2026-06-20 04:27:54 on dist-test-slave-nk6d"
server_key: "130fb43b0ecebb08308b5116486efb34"
server_key_iv: "fcf121ace544218ffbc10f663267120a"
server_key_version: "encryptionkey@0"
I20260620 04:28:37.018427 16828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:28:37.030481 16828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:28:37.033571 16828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:28:37.033749 16828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:28:37.040304 16828 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39499
I20260620 04:28:37.040545 16895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39499 every 8 connection(s)
I20260620 04:28:37.041350 16828 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:28:37.045548 16896 sys_catalog.cc:263] Verifying existing consensus state
I20260620 04:28:37.047171 16896 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cccb8275856c44bbb18eacd620970116: Bootstrap starting.
I20260620 04:28:37.051599 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 16828
I20260620 04:28:37.052114 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:34337
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:39499
--webserver_port=44573
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 04:28:37.064857 16896 log.cc:824] T 00000000000000000000000000000000 P cccb8275856c44bbb18eacd620970116: Log is configured to *not* fsync() on all Append() calls
I20260620 04:28:37.075981 16896 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cccb8275856c44bbb18eacd620970116: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 04:28:37.076318 16896 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cccb8275856c44bbb18eacd620970116: Bootstrap complete.
I20260620 04:28:37.080128 16896 raft_consensus.cc:348] T 00000000000000000000000000000000 P cccb8275856c44bbb18eacd620970116 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cccb8275856c44bbb18eacd620970116" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39499 } }
I20260620 04:28:37.080533 16896 raft_consensus.cc:729] T 00000000000000000000000000000000 P cccb8275856c44bbb18eacd620970116 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: cccb8275856c44bbb18eacd620970116, State: Initialized, Role: FOLLOWER
I20260620 04:28:37.081032 16896 consensus_queue.cc:260] T 00000000000000000000000000000000 P cccb8275856c44bbb18eacd620970116 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cccb8275856c44bbb18eacd620970116" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39499 } }
I20260620 04:28:37.081153 16896 raft_consensus.cc:388] T 00000000000000000000000000000000 P cccb8275856c44bbb18eacd620970116 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:28:37.081221 16896 raft_consensus.cc:482] T 00000000000000000000000000000000 P cccb8275856c44bbb18eacd620970116 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:28:37.081314 16896 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cccb8275856c44bbb18eacd620970116 [term 2 FOLLOWER]: Advancing to term 3
I20260620 04:28:37.083141 16896 raft_consensus.cc:504] T 00000000000000000000000000000000 P cccb8275856c44bbb18eacd620970116 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cccb8275856c44bbb18eacd620970116" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39499 } }
I20260620 04:28:37.083452 16896 leader_election.cc:302] T 00000000000000000000000000000000 P cccb8275856c44bbb18eacd620970116 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cccb8275856c44bbb18eacd620970116; no voters:
I20260620 04:28:37.083747 16896 leader_election.cc:288] T 00000000000000000000000000000000 P cccb8275856c44bbb18eacd620970116 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 04:28:37.083846 16900 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cccb8275856c44bbb18eacd620970116 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 04:28:37.084059 16900 raft_consensus.cc:686] T 00000000000000000000000000000000 P cccb8275856c44bbb18eacd620970116 [term 3 LEADER]: Becoming Leader. State: Replica: cccb8275856c44bbb18eacd620970116, State: Running, Role: LEADER
I20260620 04:28:37.084424 16900 consensus_queue.cc:237] T 00000000000000000000000000000000 P cccb8275856c44bbb18eacd620970116 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cccb8275856c44bbb18eacd620970116" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39499 } }
I20260620 04:28:37.084988 16896 sys_catalog.cc:564] T 00000000000000000000000000000000 P cccb8275856c44bbb18eacd620970116 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:28:37.085858 16901 sys_catalog.cc:455] T 00000000000000000000000000000000 P cccb8275856c44bbb18eacd620970116 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "cccb8275856c44bbb18eacd620970116" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cccb8275856c44bbb18eacd620970116" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39499 } } }
I20260620 04:28:37.085898 16902 sys_catalog.cc:455] T 00000000000000000000000000000000 P cccb8275856c44bbb18eacd620970116 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cccb8275856c44bbb18eacd620970116. Latest consensus state: current_term: 3 leader_uuid: "cccb8275856c44bbb18eacd620970116" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cccb8275856c44bbb18eacd620970116" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39499 } } }
I20260620 04:28:37.085994 16901 sys_catalog.cc:458] T 00000000000000000000000000000000 P cccb8275856c44bbb18eacd620970116 [sys.catalog]: This master's current role is: LEADER
I20260620 04:28:37.086000 16902 sys_catalog.cc:458] T 00000000000000000000000000000000 P cccb8275856c44bbb18eacd620970116 [sys.catalog]: This master's current role is: LEADER
I20260620 04:28:37.086342 16909 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:28:37.091239 16909 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:28:37.091907 16909 catalog_manager.cc:1259] Loaded cluster ID: 6722e5d138fc44cebacf726ef2c97bb5
I20260620 04:28:37.091969 16909 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:28:37.093689 16909 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:28:37.094247 16909 catalog_manager.cc:5879] T 00000000000000000000000000000000 P cccb8275856c44bbb18eacd620970116: Loaded TSK: 0
I20260620 04:28:37.094316 16909 catalog_manager.cc:5879] T 00000000000000000000000000000000 P cccb8275856c44bbb18eacd620970116: Loaded TSK: 1
F20260620 04:28:37.094873 16909 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f1aa99ef9ad google::LogMessage::Fail() at ??:0
@ 0x7f1aa99f288a google::LogMessage::SendToLog() at ??:0
@ 0x7f1aa99ef4de google::LogMessage::Flush() at ??:0
@ 0x7f1aa99f0a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f1aac6b318d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f1aac6abf0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f1ab43547bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f1ab432e99c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f1ab432f178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f1ab4370df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f1ab438f82b std::function<>::operator()() at ??:0
@ 0x7f1ab432eb4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f1ab432ff3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f1ab432c579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f1ab4370640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f1aabc413d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f1aabc41c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f1aabc433f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f1aabc32512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f1aad2766db start_thread at ??:0
@ 0x7f1aa84ed61f clone at ??:0
W20260620 04:28:37.195334 16898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:28:37.195691 16898 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:28:37.195761 16898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:28:37.201745 16898 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:28:37.201829 16898 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:28:37.201920 16898 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 04:28:37.209685 16898 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:34337
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=44573
--tserver_master_addrs=127.0.98.62:39499
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16898
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:28:37.211086 16898 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:28:37.212415 16898 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:28:37.222990 16943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:28:37.223004 16946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:28:37.223004 16944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:28:37.223498 16898 server_base.cc:1034] running on GCE node
I20260620 04:28:37.223878 16898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:28:37.224529 16898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:28:37.225697 16898 hybrid_clock.cc:648] HybridClock initialized: now 1781929717225676 us; error 36 us; skew 500 ppm
Jun 20 04:28:37 dist-test-slave-nk6d krb5kdc[15751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929717, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:28:37.228354 16898 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 04:28:37.229177 16898 webserver.cc:460] Webserver started at http://127.0.98.1:44573/ using document root <none> and password file <none>
I20260620 04:28:37.229696 16898 fs_manager.cc:359] Metadata directory not provided
I20260620 04:28:37.229751 16898 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:28:37.233863 16898 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 04:28:37.236104 16953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:28:37.237422 16898 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 04:28:37.237524 16898 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-0/wal
uuid: "54e13c6e85e747aea02cf40981ed4499"
format_stamp: "Formatted at 2026-06-20 04:27:54 on dist-test-slave-nk6d"
server_key: "cf70bf115049eb5d5d33ad49fef6187c"
server_key_iv: "b6361780b97fbbe09e78d86cb838f684"
server_key_version: "encryptionkey@0"
I20260620 04:28:37.238036 16898 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:28:37.252849 16898 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:28:37.256011 16898 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:28:37.256177 16898 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:28:37.256727 16898 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:28:37.257766 16960 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 04:28:37.263087 16898 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 04:28:37.263152 16898 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260620 04:28:37.263221 16898 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 04:28:37.265007 16898 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 04:28:37.265053 16898 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260620 04:28:37.265388 16960 tablet_bootstrap.cc:492] T df93fbb39b82405dbf56853d5de3b70d P 54e13c6e85e747aea02cf40981ed4499: Bootstrap starting.
I20260620 04:28:37.274374 16898 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34337
I20260620 04:28:37.274603 17067 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34337 every 8 connection(s)
I20260620 04:28:37.275321 16898 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-0/data/info.pb
I20260620 04:28:37.279152 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 16898
I20260620 04:28:37.279521 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:38671
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:39499
--webserver_port=43673
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 04:28:37.287794 16960 log.cc:824] T df93fbb39b82405dbf56853d5de3b70d P 54e13c6e85e747aea02cf40981ed4499: Log is configured to *not* fsync() on all Append() calls
I20260620 04:28:37.294462 16960 tablet_bootstrap.cc:492] T df93fbb39b82405dbf56853d5de3b70d P 54e13c6e85e747aea02cf40981ed4499: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 04:28:37.294760 16960 tablet_bootstrap.cc:492] T df93fbb39b82405dbf56853d5de3b70d P 54e13c6e85e747aea02cf40981ed4499: Bootstrap complete.
I20260620 04:28:37.295665 16960 ts_tablet_manager.cc:1397] T df93fbb39b82405dbf56853d5de3b70d P 54e13c6e85e747aea02cf40981ed4499: Time spent bootstrapping tablet: real 0.031s user 0.021s sys 0.004s
I20260620 04:28:37.297953 16960 raft_consensus.cc:348] T df93fbb39b82405dbf56853d5de3b70d P 54e13c6e85e747aea02cf40981ed4499 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce2d44319f6d48ee894f1d5bfc293a78" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39419 } } peers { permanent_uuid: "a546aa48667e483bbbde10dfcdf180ff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38671 } } peers { permanent_uuid: "54e13c6e85e747aea02cf40981ed4499" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34337 } }
I20260620 04:28:37.298355 16960 raft_consensus.cc:729] T df93fbb39b82405dbf56853d5de3b70d P 54e13c6e85e747aea02cf40981ed4499 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54e13c6e85e747aea02cf40981ed4499, State: Initialized, Role: FOLLOWER
I20260620 04:28:37.298794 16960 consensus_queue.cc:260] T df93fbb39b82405dbf56853d5de3b70d P 54e13c6e85e747aea02cf40981ed4499 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce2d44319f6d48ee894f1d5bfc293a78" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39419 } } peers { permanent_uuid: "a546aa48667e483bbbde10dfcdf180ff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38671 } } peers { permanent_uuid: "54e13c6e85e747aea02cf40981ed4499" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34337 } }
I20260620 04:28:37.299600 16960 ts_tablet_manager.cc:1428] T df93fbb39b82405dbf56853d5de3b70d P 54e13c6e85e747aea02cf40981ed4499: Time spent starting tablet: real 0.004s user 0.007s sys 0.000s
W20260620 04:28:37.338584 17070 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:28:37.277161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39499 (local address 127.0.98.1:51285)
0620 04:28:37.277628 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0620 04:28:37.277679 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 04:28:37.278468 (+ 789us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:28:37.337856 (+ 59388us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39499: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":118,"threads_started":1}
W20260620 04:28:37.339100 17068 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39499 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:39499: Client connection negotiation failed: client connection to 127.0.98.62:39499: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 04:28:37.339852 17070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:37.339580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39499 (local address 127.0.98.1:54299)
0620 04:28:37.339715 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 04:28:37.339778 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260620 04:28:37.340459 17070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:37.340282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39499 (local address 127.0.98.1:40861)
0620 04:28:37.340371 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0620 04:28:37.340409 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260620 04:28:37.340605 17068 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 04:28:37.406157 17071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:28:37.406415 17071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:28:37.406458 17071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:28:37.410455 17071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:28:37.410508 17071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:28:37.410565 17071 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 04:28:37.415577 17071 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:38671
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=43673
--tserver_master_addrs=127.0.98.62:39499
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17071
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:28:37.416584 17071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:28:37.417577 17071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:28:37.426666 17078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:28:37.426700 17071 server_base.cc:1034] running on GCE node
W20260620 04:28:37.426687 17081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:28:37.426668 17079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:28:37.427268 17071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:28:37.427889 17071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:28:37.429049 17071 hybrid_clock.cc:648] HybridClock initialized: now 1781929717429028 us; error 29 us; skew 500 ppm
Jun 20 04:28:37 dist-test-slave-nk6d krb5kdc[15751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929717, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:28:37.431545 17071 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 04:28:37.432399 17071 webserver.cc:460] Webserver started at http://127.0.98.2:43673/ using document root <none> and password file <none>
I20260620 04:28:37.432910 17071 fs_manager.cc:359] Metadata directory not provided
I20260620 04:28:37.432971 17071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:28:37.436609 17071 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:28:37.438540 17088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:28:37.439673 17071 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:28:37.439764 17071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-1/wal
uuid: "a546aa48667e483bbbde10dfcdf180ff"
format_stamp: "Formatted at 2026-06-20 04:27:54 on dist-test-slave-nk6d"
server_key: "417962fc6e424d7083ac63c3da11ea99"
server_key_iv: "4c9b56092034d7f31b4d4353ba22ed2b"
server_key_version: "encryptionkey@0"
I20260620 04:28:37.440315 17071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:28:37.450284 17071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:28:37.452905 17071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:28:37.453059 17071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:28:37.453619 17071 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:28:37.454398 17071 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:28:37.454450 17071 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:28:37.454509 17071 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:28:37.454537 17071 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:28:37.462520 17071 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38671
I20260620 04:28:37.462544 17201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38671 every 8 connection(s)
I20260620 04:28:37.463443 17071 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-1/data/info.pb
I20260620 04:28:37.467255 17204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:37.465614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39499 (local address 127.0.98.2:58207)
0620 04:28:37.466081 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0620 04:28:37.466569 (+ 488us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":111,"threads_started":1}
W20260620 04:28:37.467720 17202 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39499 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:39499: Client connection negotiation failed: client connection to 127.0.98.62:39499: connect: Connection refused (error 111)
I20260620 04:28:37.468309 17204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:37.468111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39499 (local address 127.0.98.2:53039)
0620 04:28:37.468209 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0620 04:28:37.468255 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260620 04:28:37.468643 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 17071
I20260620 04:28:37.468873 17204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:37.468679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39499 (local address 127.0.98.2:51409)
0620 04:28:37.468772 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0620 04:28:37.468816 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260620 04:28:37.469013 17202 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 04:28:37.469061 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:39419
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:39499
--webserver_port=44699
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:28:37.578500 17205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:28:37.578758 17205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:28:37.578816 17205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:28:37.582751 17205 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:28:37.582813 17205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:28:37.582891 17205 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 04:28:37.588055 17205 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:39419
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=44699
--tserver_master_addrs=127.0.98.62:39499
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17205
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:28:37.589072 17205 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:28:37.590072 17205 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:28:37.599507 17211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:28:37.599509 17210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:28:37.599507 17213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:28:37.599740 17205 server_base.cc:1034] running on GCE node
I20260620 04:28:37.600188 17205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:28:37.600788 17205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:28:37.601940 17205 hybrid_clock.cc:648] HybridClock initialized: now 1781929717601923 us; error 28 us; skew 500 ppm
Jun 20 04:28:37 dist-test-slave-nk6d krb5kdc[15751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929717, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:28:37.604470 17205 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 04:28:37.605268 17205 webserver.cc:460] Webserver started at http://127.0.98.3:44699/ using document root <none> and password file <none>
I20260620 04:28:37.605753 17205 fs_manager.cc:359] Metadata directory not provided
I20260620 04:28:37.605805 17205 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:28:37.609342 17205 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:28:37.611248 17220 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:28:37.612432 17205 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:28:37.612532 17205 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-2/wal
uuid: "ce2d44319f6d48ee894f1d5bfc293a78"
format_stamp: "Formatted at 2026-06-20 04:27:55 on dist-test-slave-nk6d"
server_key: "325daa7963f8f69e62f6445288a6a072"
server_key_iv: "815942cb08a3453b062f0e0623e31ad6"
server_key_version: "encryptionkey@0"
I20260620 04:28:37.613049 17205 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:28:37.623838 17205 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:28:37.626680 17205 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:28:37.626850 17205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:28:37.627384 17205 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:28:37.628180 17205 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:28:37.628225 17205 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:28:37.628276 17205 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:28:37.628300 17205 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:28:37.636925 17205 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39419
I20260620 04:28:37.636958 17333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39419 every 8 connection(s)
I20260620 04:28:37.637893 17205 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929496330398-392-0/minicluster-data/ts-2/data/info.pb
I20260620 04:28:37.641703 17336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:37.640056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39499 (local address 127.0.98.3:38807)
0620 04:28:37.640512 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0620 04:28:37.640994 (+ 482us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":123,"threads_started":1}
W20260620 04:28:37.642222 17334 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39499 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:39499: Client connection negotiation failed: client connection to 127.0.98.62:39499: connect: Connection refused (error 111)
I20260620 04:28:37.642881 17336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:37.642655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39499 (local address 127.0.98.3:34085)
0620 04:28:37.642767 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0620 04:28:37.642815 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 04:28:37.643466 17336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:37.643287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39499 (local address 127.0.98.3:60015)
0620 04:28:37.643378 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0620 04:28:37.643418 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260620 04:28:37.643635 17334 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 04:28:37.645004 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 17205
I20260620 04:28:38.341723 17338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:38.341270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39499 (local address 127.0.98.1:35777)
0620 04:28:38.341519 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0620 04:28:38.341593 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":71,"threads_started":1}
I20260620 04:28:38.469889 17339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:38.469509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39499 (local address 127.0.98.2:49371)
0620 04:28:38.469740 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0620 04:28:38.469793 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":66,"threads_started":1}
I20260620 04:28:38.644519 17340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:38.644109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39499 (local address 127.0.98.3:49457)
0620 04:28:38.644359 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0620 04:28:38.644420 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":65,"threads_started":1}
I20260620 04:28:38.774909 17341 raft_consensus.cc:482] T df93fbb39b82405dbf56853d5de3b70d P 54e13c6e85e747aea02cf40981ed4499 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:28:38.775115 17341 raft_consensus.cc:504] T df93fbb39b82405dbf56853d5de3b70d P 54e13c6e85e747aea02cf40981ed4499 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce2d44319f6d48ee894f1d5bfc293a78" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39419 } } peers { permanent_uuid: "a546aa48667e483bbbde10dfcdf180ff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38671 } } peers { permanent_uuid: "54e13c6e85e747aea02cf40981ed4499" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34337 } }
I20260620 04:28:38.776242 17341 leader_election.cc:288] T df93fbb39b82405dbf56853d5de3b70d P 54e13c6e85e747aea02cf40981ed4499 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ce2d44319f6d48ee894f1d5bfc293a78 (127.0.98.3:39419), a546aa48667e483bbbde10dfcdf180ff (127.0.98.2:38671)
Jun 20 04:28:38 dist-test-slave-nk6d krb5kdc[15751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929717, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Jun 20 04:28:38 dist-test-slave-nk6d krb5kdc[15751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929717, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260620 04:28:38.794360 17338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:38.776431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39419 (local address 127.0.98.1:33077)
0620 04:28:38.776591 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 04:28:38.776607 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 04:28:38.776713 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:28:38.778346 (+ 1633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:28:38.778351 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:28:38.778870 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:28:38.779345 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:28:38.779359 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:28:38.780692 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:28:38.780695 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:28:38.781586 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:28:38.781596 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:28:38.782260 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:28:38.782263 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:28:38.782424 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:28:38.787418 (+ 4994us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:28:38.787432 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:28:38.789233 (+ 1801us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:28:38.791638 (+ 2405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:28:38.791645 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:28:38.791661 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:28:38.792160 (+ 499us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:28:38.792512 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:28:38.792515 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:28:38.792517 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:28:38.792736 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:28:38.793475 (+ 739us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:28:38.793480 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:28:38.793711 (+ 231us) client_negotiation.cc:769] Sending connection context
0620 04:28:38.793872 (+ 161us) client_negotiation.cc:240] Negotiation successful
0620 04:28:38.794179 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":82,"spinlock_wait_cycles":37888}
I20260620 04:28:38.794450 17344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:38.776682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33077 (local address 127.0.98.3:39419)
0620 04:28:38.776935 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0620 04:28:38.776958 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0620 04:28:38.777003 (+ 45us) server_negotiation.cc:407] Connection header received
0620 04:28:38.777664 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:28:38.777670 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:28:38.778006 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:28:38.778281 (+ 275us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:28:38.779506 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:28:38.780582 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:28:38.781696 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:28:38.782154 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:28:38.782300 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:28:38.789450 (+ 7150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:28:38.789455 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:28:38.789468 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:28:38.789496 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:28:38.791499 (+ 2003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:28:38.792266 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:28:38.792272 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:28:38.792278 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:28:38.792418 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:28:38.792832 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:28:38.792835 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:28:38.792837 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:28:38.793339 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:28:38.793479 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 04:28:38.794026 (+ 547us) server_negotiation.cc:299] Negotiation successful
0620 04:28:38.794278 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":83,"threads_started":1}
I20260620 04:28:38.794634 17342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:38.776431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38671 (local address 127.0.98.1:51035)
0620 04:28:38.776999 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0620 04:28:38.777021 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 04:28:38.777123 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:28:38.778345 (+ 1222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:28:38.778350 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:28:38.778881 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:28:38.779346 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:28:38.779359 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:28:38.780681 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:28:38.780684 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:28:38.781584 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:28:38.781593 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:28:38.782552 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:28:38.782555 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:28:38.782630 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:28:38.787371 (+ 4741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:28:38.787396 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:28:38.789515 (+ 2119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:28:38.791638 (+ 2123us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:28:38.791645 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:28:38.791661 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:28:38.792155 (+ 494us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:28:38.792512 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:28:38.792515 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:28:38.792517 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:28:38.792736 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:28:38.794033 (+ 1297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:28:38.794035 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:28:38.794119 (+ 84us) client_negotiation.cc:769] Sending connection context
0620 04:28:38.794243 (+ 124us) client_negotiation.cc:240] Negotiation successful
0620 04:28:38.794358 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"mutex_wait_us":293,"spinlock_wait_cycles":24192,"thread_start_us":70,"threads_started":1}
I20260620 04:28:38.795382 17343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:38.776628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51035 (local address 127.0.98.2:38671)
0620 04:28:38.776935 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0620 04:28:38.776959 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0620 04:28:38.777106 (+ 147us) server_negotiation.cc:407] Connection header received
0620 04:28:38.777795 (+ 689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:28:38.777800 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:28:38.778096 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:28:38.778340 (+ 244us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:28:38.779503 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:28:38.780550 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:28:38.781696 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:28:38.782181 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:28:38.782692 (+ 511us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:28:38.789697 (+ 7005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:28:38.789703 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:28:38.789717 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:28:38.789751 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:28:38.791499 (+ 1748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:28:38.792266 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:28:38.792272 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:28:38.792278 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:28:38.792418 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:28:38.793547 (+ 1129us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:28:38.793550 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:28:38.793551 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:28:38.793920 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:28:38.794043 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 04:28:38.794372 (+ 329us) server_negotiation.cc:299] Negotiation successful
0620 04:28:38.795224 (+ 852us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":109,"threads_started":1}
W20260620 04:28:38.796866 16955 leader_election.cc:341] T df93fbb39b82405dbf56853d5de3b70d P 54e13c6e85e747aea02cf40981ed4499 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ce2d44319f6d48ee894f1d5bfc293a78 (127.0.98.3:39419): Not found: Tablet not found: df93fbb39b82405dbf56853d5de3b70d
W20260620 04:28:38.797108 16954 leader_election.cc:341] T df93fbb39b82405dbf56853d5de3b70d P 54e13c6e85e747aea02cf40981ed4499 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a546aa48667e483bbbde10dfcdf180ff (127.0.98.2:38671): Not found: Tablet not found: df93fbb39b82405dbf56853d5de3b70d
I20260620 04:28:38.797189 16954 leader_election.cc:302] T df93fbb39b82405dbf56853d5de3b70d P 54e13c6e85e747aea02cf40981ed4499 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 54e13c6e85e747aea02cf40981ed4499; no voters: a546aa48667e483bbbde10dfcdf180ff, ce2d44319f6d48ee894f1d5bfc293a78
I20260620 04:28:38.797426 17341 raft_consensus.cc:2726] T df93fbb39b82405dbf56853d5de3b70d P 54e13c6e85e747aea02cf40981ed4499 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 04:28:39.342905 17345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:39.342458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39499 (local address 127.0.98.1:48283)
0620 04:28:39.342738 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0620 04:28:39.342803 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":93,"threads_started":1}
I20260620 04:28:39.471016 17346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:39.470582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39499 (local address 127.0.98.2:36561)
0620 04:28:39.470860 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0620 04:28:39.470919 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":70,"threads_started":1}
I20260620 04:28:39.645478 17347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:39.645116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39499 (local address 127.0.98.3:60223)
0620 04:28:39.645342 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0620 04:28:39.645397 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":68,"threads_started":1}
I20260620 04:28:40.344013 17348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:40.343542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39499 (local address 127.0.98.1:44043)
0620 04:28:40.343819 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0620 04:28:40.343884 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":71,"threads_started":1}
I20260620 04:28:40.472213 17349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:40.471724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39499 (local address 127.0.98.2:42227)
0620 04:28:40.472017 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0620 04:28:40.472084 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":83,"threads_started":1}
I20260620 04:28:40.641147 17350 raft_consensus.cc:482] T df93fbb39b82405dbf56853d5de3b70d P 54e13c6e85e747aea02cf40981ed4499 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:28:40.641275 17350 raft_consensus.cc:504] T df93fbb39b82405dbf56853d5de3b70d P 54e13c6e85e747aea02cf40981ed4499 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce2d44319f6d48ee894f1d5bfc293a78" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39419 } } peers { permanent_uuid: "a546aa48667e483bbbde10dfcdf180ff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38671 } } peers { permanent_uuid: "54e13c6e85e747aea02cf40981ed4499" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34337 } }
I20260620 04:28:40.641708 17350 leader_election.cc:288] T df93fbb39b82405dbf56853d5de3b70d P 54e13c6e85e747aea02cf40981ed4499 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ce2d44319f6d48ee894f1d5bfc293a78 (127.0.98.3:39419), a546aa48667e483bbbde10dfcdf180ff (127.0.98.2:38671)
W20260620 04:28:40.642576 16955 leader_election.cc:341] T df93fbb39b82405dbf56853d5de3b70d P 54e13c6e85e747aea02cf40981ed4499 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ce2d44319f6d48ee894f1d5bfc293a78 (127.0.98.3:39419): Not found: Tablet not found: df93fbb39b82405dbf56853d5de3b70d
W20260620 04:28:40.642664 16954 leader_election.cc:341] T df93fbb39b82405dbf56853d5de3b70d P 54e13c6e85e747aea02cf40981ed4499 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a546aa48667e483bbbde10dfcdf180ff (127.0.98.2:38671): Not found: Tablet not found: df93fbb39b82405dbf56853d5de3b70d
I20260620 04:28:40.642750 16954 leader_election.cc:302] T df93fbb39b82405dbf56853d5de3b70d P 54e13c6e85e747aea02cf40981ed4499 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 54e13c6e85e747aea02cf40981ed4499; no voters: a546aa48667e483bbbde10dfcdf180ff, ce2d44319f6d48ee894f1d5bfc293a78
I20260620 04:28:40.642900 17350 raft_consensus.cc:2726] T df93fbb39b82405dbf56853d5de3b70d P 54e13c6e85e747aea02cf40981ed4499 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 04:28:40.646441 17351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:40.646090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39499 (local address 127.0.98.3:51577)
0620 04:28:40.646296 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0620 04:28:40.646348 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":61,"threads_started":1}
I20260620 04:28:41.345068 17352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:41.344689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39499 (local address 127.0.98.1:54595)
0620 04:28:41.344919 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0620 04:28:41.344973 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":69,"threads_started":1}
I20260620 04:28:41.473299 17353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:41.472871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39499 (local address 127.0.98.2:36137)
0620 04:28:41.473131 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0620 04:28:41.473198 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":73,"threads_started":1}
I20260620 04:28:41.647437 17354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:41.647059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39499 (local address 127.0.98.3:39229)
0620 04:28:41.647293 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0620 04:28:41.647350 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":69,"threads_started":1}
I20260620 04:28:42.346299 17355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:42.345841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39499 (local address 127.0.98.1:51453)
0620 04:28:42.346149 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0620 04:28:42.346205 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":67,"threads_started":1}
I20260620 04:28:42.474457 17356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:42.474003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39499 (local address 127.0.98.2:52017)
0620 04:28:42.474270 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0620 04:28:42.474343 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":71,"threads_started":1}
I20260620 04:28:42.648530 17357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:42.648137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39499 (local address 127.0.98.3:43351)
0620 04:28:42.648371 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0620 04:28:42.648434 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":75,"threads_started":1}
I20260620 04:28:42.896795 17358 raft_consensus.cc:482] T df93fbb39b82405dbf56853d5de3b70d P 54e13c6e85e747aea02cf40981ed4499 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:28:42.896931 17358 raft_consensus.cc:504] T df93fbb39b82405dbf56853d5de3b70d P 54e13c6e85e747aea02cf40981ed4499 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce2d44319f6d48ee894f1d5bfc293a78" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39419 } } peers { permanent_uuid: "a546aa48667e483bbbde10dfcdf180ff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38671 } } peers { permanent_uuid: "54e13c6e85e747aea02cf40981ed4499" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34337 } }
I20260620 04:28:42.897328 17358 leader_election.cc:288] T df93fbb39b82405dbf56853d5de3b70d P 54e13c6e85e747aea02cf40981ed4499 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ce2d44319f6d48ee894f1d5bfc293a78 (127.0.98.3:39419), a546aa48667e483bbbde10dfcdf180ff (127.0.98.2:38671)
W20260620 04:28:42.898026 16955 leader_election.cc:341] T df93fbb39b82405dbf56853d5de3b70d P 54e13c6e85e747aea02cf40981ed4499 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ce2d44319f6d48ee894f1d5bfc293a78 (127.0.98.3:39419): Not found: Tablet not found: df93fbb39b82405dbf56853d5de3b70d
W20260620 04:28:42.898129 16954 leader_election.cc:341] T df93fbb39b82405dbf56853d5de3b70d P 54e13c6e85e747aea02cf40981ed4499 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a546aa48667e483bbbde10dfcdf180ff (127.0.98.2:38671): Not found: Tablet not found: df93fbb39b82405dbf56853d5de3b70d
I20260620 04:28:42.898216 16954 leader_election.cc:302] T df93fbb39b82405dbf56853d5de3b70d P 54e13c6e85e747aea02cf40981ed4499 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 54e13c6e85e747aea02cf40981ed4499; no voters: a546aa48667e483bbbde10dfcdf180ff, ce2d44319f6d48ee894f1d5bfc293a78
I20260620 04:28:42.898342 17358 raft_consensus.cc:2726] T df93fbb39b82405dbf56853d5de3b70d P 54e13c6e85e747aea02cf40981ed4499 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 04:28:43.347621 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:43.347197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39499 (local address 127.0.98.1:44059)
0620 04:28:43.347463 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0620 04:28:43.347526 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":77,"threads_started":1}
I20260620 04:28:43.475864 17360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:43.475410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39499 (local address 127.0.98.2:47477)
0620 04:28:43.475688 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0620 04:28:43.475758 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":76,"threads_started":1}
I20260620 04:28:43.649652 17361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:43.649273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39499 (local address 127.0.98.3:36333)
0620 04:28:43.649501 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0620 04:28:43.649559 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":68,"threads_started":1}
I20260620 04:28:44.348737 17362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:44.348353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39499 (local address 127.0.98.1:58633)
0620 04:28:44.348586 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0620 04:28:44.348640 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":71,"threads_started":1}
I20260620 04:28:44.477094 17363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:44.476630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39499 (local address 127.0.98.2:40967)
0620 04:28:44.476944 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0620 04:28:44.477002 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":73,"threads_started":1}
I20260620 04:28:44.650763 17364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:44.650291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39499 (local address 127.0.98.3:33581)
0620 04:28:44.650596 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0620 04:28:44.650671 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":78,"threads_started":1}
I20260620 04:28:45.349922 17365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:45.349511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39499 (local address 127.0.98.1:49651)
0620 04:28:45.349766 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0620 04:28:45.349826 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":71,"threads_started":1}
I20260620 04:28:45.478266 17366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:45.477847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39499 (local address 127.0.98.2:56729)
0620 04:28:45.478106 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0620 04:28:45.478164 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":72,"threads_started":1}
I20260620 04:28:45.651813 17367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:45.651421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39499 (local address 127.0.98.3:34939)
0620 04:28:45.651663 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0620 04:28:45.651721 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":65,"threads_started":1}
I20260620 04:28:46.351155 17368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:46.350670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39499 (local address 127.0.98.1:39703)
0620 04:28:46.350949 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0620 04:28:46.351028 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":69,"threads_started":1}
I20260620 04:28:46.479422 17369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:46.479020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39499 (local address 127.0.98.2:53685)
0620 04:28:46.479264 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0620 04:28:46.479327 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":73,"threads_started":1}
I20260620 04:28:46.652988 17370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:46.652547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39499 (local address 127.0.98.3:36527)
0620 04:28:46.652815 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0620 04:28:46.652893 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":78,"threads_started":1}
I20260620 04:28:47.272801 17371 raft_consensus.cc:482] T df93fbb39b82405dbf56853d5de3b70d P 54e13c6e85e747aea02cf40981ed4499 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:28:47.272948 17371 raft_consensus.cc:504] T df93fbb39b82405dbf56853d5de3b70d P 54e13c6e85e747aea02cf40981ed4499 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce2d44319f6d48ee894f1d5bfc293a78" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39419 } } peers { permanent_uuid: "a546aa48667e483bbbde10dfcdf180ff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38671 } } peers { permanent_uuid: "54e13c6e85e747aea02cf40981ed4499" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34337 } }
I20260620 04:28:47.273355 17371 leader_election.cc:288] T df93fbb39b82405dbf56853d5de3b70d P 54e13c6e85e747aea02cf40981ed4499 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ce2d44319f6d48ee894f1d5bfc293a78 (127.0.98.3:39419), a546aa48667e483bbbde10dfcdf180ff (127.0.98.2:38671)
W20260620 04:28:47.274122 16955 leader_election.cc:341] T df93fbb39b82405dbf56853d5de3b70d P 54e13c6e85e747aea02cf40981ed4499 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ce2d44319f6d48ee894f1d5bfc293a78 (127.0.98.3:39419): Not found: Tablet not found: df93fbb39b82405dbf56853d5de3b70d
W20260620 04:28:47.274245 16954 leader_election.cc:341] T df93fbb39b82405dbf56853d5de3b70d P 54e13c6e85e747aea02cf40981ed4499 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a546aa48667e483bbbde10dfcdf180ff (127.0.98.2:38671): Not found: Tablet not found: df93fbb39b82405dbf56853d5de3b70d
I20260620 04:28:47.274339 16954 leader_election.cc:302] T df93fbb39b82405dbf56853d5de3b70d P 54e13c6e85e747aea02cf40981ed4499 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 54e13c6e85e747aea02cf40981ed4499; no voters: a546aa48667e483bbbde10dfcdf180ff, ce2d44319f6d48ee894f1d5bfc293a78
I20260620 04:28:47.274482 17371 raft_consensus.cc:2726] T df93fbb39b82405dbf56853d5de3b70d P 54e13c6e85e747aea02cf40981ed4499 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 04:28:47.352383 17372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:47.352014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39499 (local address 127.0.98.1:46245)
0620 04:28:47.352241 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0620 04:28:47.352291 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":72,"threads_started":1}
I20260620 04:28:47.480726 17373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:47.480322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39499 (local address 127.0.98.2:53251)
0620 04:28:47.480569 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0620 04:28:47.480632 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":113,"threads_started":1}
I20260620 04:28:47.654098 17374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:47.653700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39499 (local address 127.0.98.3:55471)
0620 04:28:47.653934 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0620 04:28:47.653993 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":66,"threads_started":1}
I20260620 04:28:48.353466 17375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:48.353071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39499 (local address 127.0.98.1:41191)
0620 04:28:48.353308 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0620 04:28:48.353369 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":76,"threads_started":1}
I20260620 04:28:48.481886 17376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:48.481427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39499 (local address 127.0.98.2:40625)
0620 04:28:48.481716 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0620 04:28:48.481779 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":73,"threads_started":1}
I20260620 04:28:48.655171 17377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:48.654793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39499 (local address 127.0.98.3:33397)
0620 04:28:48.655013 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0620 04:28:48.655070 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":67,"threads_started":1}
I20260620 04:28:49.354719 17378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:49.354290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39499 (local address 127.0.98.1:54301)
0620 04:28:49.354546 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0620 04:28:49.354614 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":81,"threads_started":1}
I20260620 04:28:49.483265 17379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:49.482776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39499 (local address 127.0.98.2:51413)
0620 04:28:49.483098 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0620 04:28:49.483168 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":87,"threads_started":1}
I20260620 04:28:49.656315 17380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:49.655880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39499 (local address 127.0.98.3:49811)
0620 04:28:49.656161 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0620 04:28:49.656223 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":121,"threads_started":1}
I20260620 04:28:50.355877 17381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:50.355393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39499 (local address 127.0.98.1:33947)
0620 04:28:50.355681 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0620 04:28:50.355742 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":77,"threads_started":1}
I20260620 04:28:50.484472 17382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:50.484072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39499 (local address 127.0.98.2:56221)
0620 04:28:50.484315 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0620 04:28:50.484378 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":68,"threads_started":1}
I20260620 04:28:50.657351 17383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:50.656953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39499 (local address 127.0.98.3:58455)
0620 04:28:50.657196 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0620 04:28:50.657251 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":71,"threads_started":1}
I20260620 04:28:50.755584 17384 raft_consensus.cc:482] T df93fbb39b82405dbf56853d5de3b70d P 54e13c6e85e747aea02cf40981ed4499 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:28:50.755708 17384 raft_consensus.cc:504] T df93fbb39b82405dbf56853d5de3b70d P 54e13c6e85e747aea02cf40981ed4499 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce2d44319f6d48ee894f1d5bfc293a78" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39419 } } peers { permanent_uuid: "a546aa48667e483bbbde10dfcdf180ff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38671 } } peers { permanent_uuid: "54e13c6e85e747aea02cf40981ed4499" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34337 } }
I20260620 04:28:50.756109 17384 leader_election.cc:288] T df93fbb39b82405dbf56853d5de3b70d P 54e13c6e85e747aea02cf40981ed4499 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ce2d44319f6d48ee894f1d5bfc293a78 (127.0.98.3:39419), a546aa48667e483bbbde10dfcdf180ff (127.0.98.2:38671)
W20260620 04:28:50.756765 16954 leader_election.cc:341] T df93fbb39b82405dbf56853d5de3b70d P 54e13c6e85e747aea02cf40981ed4499 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a546aa48667e483bbbde10dfcdf180ff (127.0.98.2:38671): Not found: Tablet not found: df93fbb39b82405dbf56853d5de3b70d
W20260620 04:28:50.756875 16955 leader_election.cc:341] T df93fbb39b82405dbf56853d5de3b70d P 54e13c6e85e747aea02cf40981ed4499 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ce2d44319f6d48ee894f1d5bfc293a78 (127.0.98.3:39419): Not found: Tablet not found: df93fbb39b82405dbf56853d5de3b70d
I20260620 04:28:50.756950 16955 leader_election.cc:302] T df93fbb39b82405dbf56853d5de3b70d P 54e13c6e85e747aea02cf40981ed4499 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 54e13c6e85e747aea02cf40981ed4499; no voters: a546aa48667e483bbbde10dfcdf180ff, ce2d44319f6d48ee894f1d5bfc293a78
I20260620 04:28:50.757082 17384 raft_consensus.cc:2726] T df93fbb39b82405dbf56853d5de3b70d P 54e13c6e85e747aea02cf40981ed4499 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 04:28:51.356992 17385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:51.356596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39499 (local address 127.0.98.1:55065)
0620 04:28:51.356837 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0620 04:28:51.356894 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":83,"threads_started":1}
I20260620 04:28:51.485591 17386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:51.485182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39499 (local address 127.0.98.2:37269)
0620 04:28:51.485429 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0620 04:28:51.485488 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":85,"threads_started":1}
I20260620 04:28:51.658445 17387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:51.658041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39499 (local address 127.0.98.3:46773)
0620 04:28:51.658293 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0620 04:28:51.658353 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":77,"threads_started":1}
I20260620 04:28:52.358143 17388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:52.357802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39499 (local address 127.0.98.1:58319)
0620 04:28:52.358003 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0620 04:28:52.358052 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":61,"threads_started":1}
I20260620 04:28:52.486846 17389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:52.486431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39499 (local address 127.0.98.2:51397)
0620 04:28:52.486701 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0620 04:28:52.486753 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":72,"threads_started":1}
I20260620 04:28:52.648190 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 16898
I20260620 04:28:52.653270 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 17071
I20260620 04:28:52.657620 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 17205
2026-06-20T04:28:52Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59329 ms)
[----------] 30 tests from SecurityITest (236319 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929496330398-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:28:52 dist-test-slave-nk6d krb5kdc[17392](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:28:52 dist-test-slave-nk6d krb5kdc[17392](info): set up 2 sockets
Jun 20 04:28:52 dist-test-slave-nk6d krb5kdc[17392](info): commencing operation
krb5kdc: starting...
W20260620 04:28:53.615576 392 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929496330398-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929496330398-392-0/krb5kdc/test-user.keytab.
Jun 20 04:28:53 dist-test-slave-nk6d krb5kdc[17392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929733, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:28:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:28:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 04:28:53.723249 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929496330398-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929496330398-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929496330398-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36651
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36651
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:28:53.829839 17405 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:28:53.830114 17405 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:28:53.830154 17405 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:28:53.834153 17405 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:28:53.834218 17405 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:28:53.834237 17405 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:28:53.834251 17405 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:28:53.834265 17405 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:28:53.839414 17405 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929496330398-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929496330398-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36651
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36651
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17405
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929496330398-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:28:53.840425 17405 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:28:53.841357 17405 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 04:28:53.849669 17405 server_base.cc:1034] running on GCE node
W20260620 04:28:53.849675 17410 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:28:53.849686 17413 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:28:53.849697 17411 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:28:53.850513 17405 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:28:53.851580 17405 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:28:53.852722 17405 hybrid_clock.cc:648] HybridClock initialized: now 1781929733852703 us; error 32 us; skew 500 ppm
Jun 20 04:28:53 dist-test-slave-nk6d krb5kdc[17392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929733, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:28:53.855274 17405 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 04:28:53.856134 17405 webserver.cc:460] Webserver started at http://127.0.98.62:42475/ using document root <none> and password file <none>
I20260620 04:28:53.856642 17405 fs_manager.cc:359] Metadata directory not provided
I20260620 04:28:53.856698 17405 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:28:53.856856 17405 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:28:53.858786 17405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929496330398-392-0/minicluster-data/master-0/data/instance:
uuid: "eb6f5a8737fe4270867db8d990ae75c3"
format_stamp: "Formatted at 2026-06-20 04:28:53 on dist-test-slave-nk6d"
server_key: "41d9c8184a192d59281ee3ad9afd953d"
server_key_iv: "0f048d172f26b3a14f59e49cb41d4e4f"
server_key_version: "encryptionkey@0"
I20260620 04:28:53.859258 17405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929496330398-392-0/minicluster-data/master-0/wal/instance:
uuid: "eb6f5a8737fe4270867db8d990ae75c3"
format_stamp: "Formatted at 2026-06-20 04:28:53 on dist-test-slave-nk6d"
server_key: "41d9c8184a192d59281ee3ad9afd953d"
server_key_iv: "0f048d172f26b3a14f59e49cb41d4e4f"
server_key_version: "encryptionkey@0"
I20260620 04:28:53.862655 17405 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 04:28:53.864890 17420 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:28:53.866027 17405 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:28:53.866138 17405 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929496330398-392-0/minicluster-data/master-0/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929496330398-392-0/minicluster-data/master-0/wal
uuid: "eb6f5a8737fe4270867db8d990ae75c3"
format_stamp: "Formatted at 2026-06-20 04:28:53 on dist-test-slave-nk6d"
server_key: "41d9c8184a192d59281ee3ad9afd953d"
server_key_iv: "0f048d172f26b3a14f59e49cb41d4e4f"
server_key_version: "encryptionkey@0"
I20260620 04:28:53.866225 17405 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929496330398-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929496330398-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929496330398-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:28:53.875633 17405 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:28:53.878190 17405 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:28:53.878343 17405 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:28:53.884826 17405 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36651
I20260620 04:28:53.884845 17472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36651 every 8 connection(s)
I20260620 04:28:53.885720 17405 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:28:53.888254 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 17405
I20260620 04:28:53.888377 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929496330398-392-0/minicluster-data/master-0/wal/instance
I20260620 04:28:53.888480 17473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:28:53.888669 392 external_mini_cluster.cc:1428] Setting key 6bf3e232603307730234c987b0d7bf17
I20260620 04:28:53.894642 17473 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb6f5a8737fe4270867db8d990ae75c3: Bootstrap starting.
I20260620 04:28:53.896852 17473 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eb6f5a8737fe4270867db8d990ae75c3: Neither blocks nor log segments found. Creating new log.
Jun 20 04:28:53 dist-test-slave-nk6d krb5kdc[17392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929733, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:28:53.897535 17473 log.cc:824] T 00000000000000000000000000000000 P eb6f5a8737fe4270867db8d990ae75c3: Log is configured to *not* fsync() on all Append() calls
I20260620 04:28:53.899587 17473 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb6f5a8737fe4270867db8d990ae75c3: No bootstrap required, opened a new log
I20260620 04:28:53.902156 17476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:53.890019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37422 (local address 127.0.98.62:36651)
0620 04:28:53.890434 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0620 04:28:53.890448 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 04:28:53.890482 (+ 34us) server_negotiation.cc:407] Connection header received
0620 04:28:53.891084 (+ 602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:28:53.891100 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:28:53.891409 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:28:53.891705 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:28:53.892532 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:28:53.893507 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:28:53.894343 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:28:53.894953 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:28:53.895095 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:28:53.897390 (+ 2295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:28:53.897396 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:28:53.897408 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:28:53.897434 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:28:53.899268 (+ 1834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:28:53.899688 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:28:53.899694 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:28:53.899699 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:28:53.899832 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:28:53.900104 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:28:53.900107 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:28:53.900109 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:28:53.900573 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:28:53.900751 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0620 04:28:53.900945 (+ 194us) server_negotiation.cc:299] Negotiation successful
0620 04:28:53.901219 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":131,"threads_started":1}
I20260620 04:28:53.903533 17473 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb6f5a8737fe4270867db8d990ae75c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb6f5a8737fe4270867db8d990ae75c3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36651 } }
I20260620 04:28:53.903739 17473 raft_consensus.cc:374] T 00000000000000000000000000000000 P eb6f5a8737fe4270867db8d990ae75c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:28:53.903774 17473 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb6f5a8737fe4270867db8d990ae75c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb6f5a8737fe4270867db8d990ae75c3, State: Initialized, Role: FOLLOWER
I20260620 04:28:53.904196 17473 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb6f5a8737fe4270867db8d990ae75c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb6f5a8737fe4270867db8d990ae75c3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36651 } }
I20260620 04:28:53.904323 17473 raft_consensus.cc:388] T 00000000000000000000000000000000 P eb6f5a8737fe4270867db8d990ae75c3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:28:53.904399 17473 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb6f5a8737fe4270867db8d990ae75c3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:28:53.904496 17473 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb6f5a8737fe4270867db8d990ae75c3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:28:53.905282 17473 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb6f5a8737fe4270867db8d990ae75c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb6f5a8737fe4270867db8d990ae75c3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36651 } }
I20260620 04:28:53.905587 17473 leader_election.cc:302] T 00000000000000000000000000000000 P eb6f5a8737fe4270867db8d990ae75c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eb6f5a8737fe4270867db8d990ae75c3; no voters:
I20260620 04:28:53.905856 17473 leader_election.cc:288] T 00000000000000000000000000000000 P eb6f5a8737fe4270867db8d990ae75c3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:28:53.905966 17478 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eb6f5a8737fe4270867db8d990ae75c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:28:53.906163 17478 raft_consensus.cc:686] T 00000000000000000000000000000000 P eb6f5a8737fe4270867db8d990ae75c3 [term 1 LEADER]: Becoming Leader. State: Replica: eb6f5a8737fe4270867db8d990ae75c3, State: Running, Role: LEADER
I20260620 04:28:53.906507 17478 consensus_queue.cc:237] T 00000000000000000000000000000000 P eb6f5a8737fe4270867db8d990ae75c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb6f5a8737fe4270867db8d990ae75c3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36651 } }
I20260620 04:28:53.906963 17473 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb6f5a8737fe4270867db8d990ae75c3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:28:53.908010 17480 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb6f5a8737fe4270867db8d990ae75c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb6f5a8737fe4270867db8d990ae75c3. Latest consensus state: current_term: 1 leader_uuid: "eb6f5a8737fe4270867db8d990ae75c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb6f5a8737fe4270867db8d990ae75c3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36651 } } }
I20260620 04:28:53.908118 17480 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb6f5a8737fe4270867db8d990ae75c3 [sys.catalog]: This master's current role is: LEADER
I20260620 04:28:53.908321 17479 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb6f5a8737fe4270867db8d990ae75c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eb6f5a8737fe4270867db8d990ae75c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb6f5a8737fe4270867db8d990ae75c3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36651 } } }
I20260620 04:28:53.908404 17479 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb6f5a8737fe4270867db8d990ae75c3 [sys.catalog]: This master's current role is: LEADER
I20260620 04:28:53.908672 17487 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:28:53.911412 17487 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:28:53.916446 17487 catalog_manager.cc:1347] Generated new cluster ID: c1c263b9b0b140a095f159ca948f6d7c
I20260620 04:28:53.916514 17487 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:28:53.931883 17487 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:28:53.932818 17487 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:28:53.943917 17487 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eb6f5a8737fe4270867db8d990ae75c3: Generated new TSK 0
I20260620 04:28:53.944607 17487 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929496330398-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929496330398-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 04:28:53.976900 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929496330398-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929496330398-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929496330398-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929496330398-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36651
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929496330398-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:28:54.108225 17499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:28:54.108564 17499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:28:54.108625 17499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:28:54.112656 17499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:28:54.112716 17499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:28:54.112795 17499 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 04:28:54.118044 17499 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929496330398-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929496330398-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929496330398-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929496330398-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36651
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17499
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929496330398-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:28:54.119072 17499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:28:54.120097 17499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:28:54.129623 17504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:28:54.129701 17505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:28:54.129843 17499 server_base.cc:1034] running on GCE node
W20260620 04:28:54.129612 17507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:28:54.130283 17499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:28:54.130898 17499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:28:54.132035 17499 hybrid_clock.cc:648] HybridClock initialized: now 1781929734132015 us; error 30 us; skew 500 ppm
Jun 20 04:28:54 dist-test-slave-nk6d krb5kdc[17392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929734, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:28:54.134640 17499 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 04:28:54.135483 17499 webserver.cc:460] Webserver started at http://127.0.98.1:46201/ using document root <none> and password file <none>
I20260620 04:28:54.136041 17499 fs_manager.cc:359] Metadata directory not provided
I20260620 04:28:54.136109 17499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:28:54.136296 17499 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:28:54.138254 17499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929496330398-392-0/minicluster-data/ts-0/data/instance:
uuid: "edaf882aed364cb29537e333e87901b3"
format_stamp: "Formatted at 2026-06-20 04:28:54 on dist-test-slave-nk6d"
server_key: "d32cfdb6eee1ecf7a55c877351a155cd"
server_key_iv: "0ea8dca253ab736207fc076870f566b1"
server_key_version: "encryptionkey@0"
I20260620 04:28:54.138782 17499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929496330398-392-0/minicluster-data/ts-0/wal/instance:
uuid: "edaf882aed364cb29537e333e87901b3"
format_stamp: "Formatted at 2026-06-20 04:28:54 on dist-test-slave-nk6d"
server_key: "d32cfdb6eee1ecf7a55c877351a155cd"
server_key_iv: "0ea8dca253ab736207fc076870f566b1"
server_key_version: "encryptionkey@0"
I20260620 04:28:54.142283 17499 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260620 04:28:54.144699 17514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:28:54.145891 17499 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:28:54.145992 17499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929496330398-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929496330398-392-0/minicluster-data/ts-0/wal
uuid: "edaf882aed364cb29537e333e87901b3"
format_stamp: "Formatted at 2026-06-20 04:28:54 on dist-test-slave-nk6d"
server_key: "d32cfdb6eee1ecf7a55c877351a155cd"
server_key_iv: "0ea8dca253ab736207fc076870f566b1"
server_key_version: "encryptionkey@0"
I20260620 04:28:54.146080 17499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929496330398-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929496330398-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929496330398-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:28:54.162317 17499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:28:54.165591 17499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:28:54.165763 17499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:28:54.166337 17499 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:28:54.167157 17499 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:28:54.167202 17499 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:28:54.167254 17499 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:28:54.167279 17499 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:28:54.175649 17499 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34241
I20260620 04:28:54.175670 17627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34241 every 8 connection(s)
I20260620 04:28:54.176573 17499 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929496330398-392-0/minicluster-data/ts-0/data/info.pb
I20260620 04:28:54.183486 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 17499
I20260620 04:28:54.183624 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929496330398-392-0/minicluster-data/ts-0/wal/instance
I20260620 04:28:54.183890 392 external_mini_cluster.cc:1428] Setting key f906d79cc4cbc6dd8f76ad597b8b7fe7
Jun 20 04:28:54 dist-test-slave-nk6d krb5kdc[17392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929734, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:28:54.190738 17476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:54.178316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33299 (local address 127.0.98.62:36651)
0620 04:28:54.178468 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 04:28:54.178473 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:28:54.179376 (+ 903us) server_negotiation.cc:407] Connection header received
0620 04:28:54.180641 (+ 1265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:28:54.180644 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:28:54.180695 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:28:54.180774 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:28:54.181889 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:28:54.182421 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:28:54.183603 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:28:54.183918 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:28:54.184078 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:28:54.186843 (+ 2765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:28:54.186848 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:28:54.186851 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:28:54.186881 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:28:54.188680 (+ 1799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:28:54.189178 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:28:54.189183 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:28:54.189185 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:28:54.189295 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:28:54.189604 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:28:54.189607 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:28:54.189609 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:28:54.189864 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:28:54.189985 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 04:28:54.190409 (+ 424us) server_negotiation.cc:299] Negotiation successful
0620 04:28:54.190545 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 04:28:54.191433 17630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:54.178775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36651 (local address 127.0.98.1:33299)
0620 04:28:54.179223 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0620 04:28:54.179267 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 04:28:54.180424 (+ 1157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:28:54.180939 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:28:54.180945 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:28:54.181288 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:28:54.181756 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:28:54.181769 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:28:54.182513 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:28:54.182516 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:28:54.183452 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:28:54.183462 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:28:54.184114 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:28:54.184118 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:28:54.184333 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:28:54.184981 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:28:54.185005 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:28:54.186637 (+ 1632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:28:54.188790 (+ 2153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:28:54.188794 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:28:54.188805 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:28:54.189071 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:28:54.189383 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:28:54.189385 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:28:54.189386 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:28:54.189519 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:28:54.189961 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:28:54.189966 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:28:54.190184 (+ 218us) client_negotiation.cc:769] Sending connection context
0620 04:28:54.190329 (+ 145us) client_negotiation.cc:240] Negotiation successful
0620 04:28:54.190567 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":112,"threads_started":1}
I20260620 04:28:54.192685 17628 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36651
I20260620 04:28:54.192925 17628 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 04:28:54.193385 17628 heartbeater.cc:511] Master 127.0.98.62:36651 requested a full tablet report, sending...
I20260620 04:28:54.194800 17437 ts_manager.cc:194] Registered new tserver with Master: edaf882aed364cb29537e333e87901b3 (127.0.98.1:34241)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 04:28:54.196159 17437 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:33299
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929496330398-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929496330398-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 04:28:54.208899 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929496330398-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929496330398-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929496330398-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929496330398-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36651
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929496330398-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:28:54.313490 17633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:28:54.313746 17633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:28:54.313805 17633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:28:54.317716 17633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:28:54.317771 17633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:28:54.317850 17633 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 04:28:54.322918 17633 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929496330398-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929496330398-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929496330398-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929496330398-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36651
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17633
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929496330398-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:28:54.323944 17633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:28:54.324956 17633 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:28:54.333951 17638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:28:54.333951 17639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:28:54.333997 17641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:28:54.334146 17633 server_base.cc:1034] running on GCE node
I20260620 04:28:54.334761 17633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:28:54.335358 17633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:28:54.336505 17633 hybrid_clock.cc:648] HybridClock initialized: now 1781929734336485 us; error 30 us; skew 500 ppm
Jun 20 04:28:54 dist-test-slave-nk6d krb5kdc[17392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929734, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:28:54.339037 17633 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 04:28:54.339883 17633 webserver.cc:460] Webserver started at http://127.0.98.2:42611/ using document root <none> and password file <none>
I20260620 04:28:54.340417 17633 fs_manager.cc:359] Metadata directory not provided
I20260620 04:28:54.340480 17633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:28:54.340667 17633 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:28:54.342576 17633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929496330398-392-0/minicluster-data/ts-1/data/instance:
uuid: "4f34bb65e7be435c97df5d178f9d179f"
format_stamp: "Formatted at 2026-06-20 04:28:54 on dist-test-slave-nk6d"
server_key: "17af4e7e2778c2317d37cea684ca4b69"
server_key_iv: "68577d091be321208a2586a4c9a77d54"
server_key_version: "encryptionkey@0"
I20260620 04:28:54.343086 17633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929496330398-392-0/minicluster-data/ts-1/wal/instance:
uuid: "4f34bb65e7be435c97df5d178f9d179f"
format_stamp: "Formatted at 2026-06-20 04:28:54 on dist-test-slave-nk6d"
server_key: "17af4e7e2778c2317d37cea684ca4b69"
server_key_iv: "68577d091be321208a2586a4c9a77d54"
server_key_version: "encryptionkey@0"
I20260620 04:28:54.346526 17633 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260620 04:28:54.348781 17648 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:28:54.349907 17633 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260620 04:28:54.350008 17633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929496330398-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929496330398-392-0/minicluster-data/ts-1/wal
uuid: "4f34bb65e7be435c97df5d178f9d179f"
format_stamp: "Formatted at 2026-06-20 04:28:54 on dist-test-slave-nk6d"
server_key: "17af4e7e2778c2317d37cea684ca4b69"
server_key_iv: "68577d091be321208a2586a4c9a77d54"
server_key_version: "encryptionkey@0"
I20260620 04:28:54.350090 17633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929496330398-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929496330398-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929496330398-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:28:54.367492 17633 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:28:54.370129 17633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:28:54.370309 17633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:28:54.370888 17633 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:28:54.371645 17633 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:28:54.371690 17633 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:28:54.371742 17633 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:28:54.371769 17633 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:28:54.379786 17633 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46339
I20260620 04:28:54.379810 17761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46339 every 8 connection(s)
I20260620 04:28:54.380658 17633 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929496330398-392-0/minicluster-data/ts-1/data/info.pb
I20260620 04:28:54.384300 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 17633
I20260620 04:28:54.384425 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929496330398-392-0/minicluster-data/ts-1/wal/instance
I20260620 04:28:54.384714 392 external_mini_cluster.cc:1428] Setting key 3d8564540d52e81b571de48caee06143
Jun 20 04:28:54 dist-test-slave-nk6d krb5kdc[17392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929734, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 04:28:54.394352 17476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:54.382339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42613 (local address 127.0.98.62:36651)
0620 04:28:54.382486 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 04:28:54.382489 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:28:54.383246 (+ 757us) server_negotiation.cc:407] Connection header received
0620 04:28:54.384463 (+ 1217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:28:54.384466 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:28:54.384520 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:28:54.384593 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:28:54.385912 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:28:54.386621 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:28:54.388100 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:28:54.388427 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:28:54.388538 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:28:54.390957 (+ 2419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:28:54.390961 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:28:54.390964 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:28:54.390993 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:28:54.392527 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:28:54.392976 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:28:54.392980 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:28:54.392982 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:28:54.393069 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:28:54.393346 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:28:54.393349 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:28:54.393351 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:28:54.393568 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:28:54.393674 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 04:28:54.394125 (+ 451us) server_negotiation.cc:299] Negotiation successful
0620 04:28:54.394234 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 04:28:54.395154 17764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:54.382700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36651 (local address 127.0.98.2:42613)
0620 04:28:54.383105 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0620 04:28:54.383144 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 04:28:54.384265 (+ 1121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:28:54.384819 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:28:54.384828 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:28:54.385260 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:28:54.385764 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:28:54.385782 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:28:54.386772 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:28:54.386775 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:28:54.387990 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:28:54.388001 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:28:54.388552 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:28:54.388554 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:28:54.388683 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:28:54.389277 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:28:54.389293 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:28:54.390792 (+ 1499us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:28:54.392634 (+ 1842us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:28:54.392638 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:28:54.392650 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:28:54.392882 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:28:54.393150 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:28:54.393152 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:28:54.393153 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:28:54.393266 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:28:54.393661 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:28:54.393666 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:28:54.393884 (+ 218us) client_negotiation.cc:769] Sending connection context
0620 04:28:54.394061 (+ 177us) client_negotiation.cc:240] Negotiation successful
0620 04:28:54.394309 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":102,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260620 04:28:54.396265 17762 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36651
I20260620 04:28:54.396518 17762 heartbeater.cc:464] Registering TS with master...
I20260620 04:28:54.397156 17762 heartbeater.cc:511] Master 127.0.98.62:36651 requested a full tablet report, sending...
I20260620 04:28:54.398234 17437 ts_manager.cc:194] Registered new tserver with Master: 4f34bb65e7be435c97df5d178f9d179f (127.0.98.2:46339)
I20260620 04:28:54.398788 17437 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42613
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929496330398-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929496330398-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 04:28:54.410574 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929496330398-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929496330398-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929496330398-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929496330398-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36651
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929496330398-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:28:54.515787 17767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:28:54.516059 17767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:28:54.516100 17767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:28:54.519934 17767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:28:54.519999 17767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:28:54.520054 17767 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 04:28:54.525082 17767 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929496330398-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929496330398-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929496330398-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929496330398-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36651
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17767
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929496330398-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:28:54.526023 17767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:28:54.527055 17767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:28:54.536741 17773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:28:54.536736 17775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:28:54.536736 17772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:28:54.537137 17767 server_base.cc:1034] running on GCE node
I20260620 04:28:54.537477 17767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:28:54.538021 17767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:28:54.539182 17767 hybrid_clock.cc:648] HybridClock initialized: now 1781929734539153 us; error 35 us; skew 500 ppm
Jun 20 04:28:54 dist-test-slave-nk6d krb5kdc[17392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929734, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:28:54.541733 17767 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 04:28:54.542526 17767 webserver.cc:460] Webserver started at http://127.0.98.3:33421/ using document root <none> and password file <none>
I20260620 04:28:54.543006 17767 fs_manager.cc:359] Metadata directory not provided
I20260620 04:28:54.543058 17767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:28:54.543216 17767 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:28:54.545140 17767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929496330398-392-0/minicluster-data/ts-2/data/instance:
uuid: "30b82a1886514cb7b07a4d1349d86fcd"
format_stamp: "Formatted at 2026-06-20 04:28:54 on dist-test-slave-nk6d"
server_key: "8e91a38a6938cda161f39ecf4cf17328"
server_key_iv: "7794593386b81652fcd1e0c9ccb37877"
server_key_version: "encryptionkey@0"
I20260620 04:28:54.545676 17767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929496330398-392-0/minicluster-data/ts-2/wal/instance:
uuid: "30b82a1886514cb7b07a4d1349d86fcd"
format_stamp: "Formatted at 2026-06-20 04:28:54 on dist-test-slave-nk6d"
server_key: "8e91a38a6938cda161f39ecf4cf17328"
server_key_iv: "7794593386b81652fcd1e0c9ccb37877"
server_key_version: "encryptionkey@0"
I20260620 04:28:54.549129 17767 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 04:28:54.551465 17782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:28:54.552678 17767 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:28:54.552773 17767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929496330398-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929496330398-392-0/minicluster-data/ts-2/wal
uuid: "30b82a1886514cb7b07a4d1349d86fcd"
format_stamp: "Formatted at 2026-06-20 04:28:54 on dist-test-slave-nk6d"
server_key: "8e91a38a6938cda161f39ecf4cf17328"
server_key_iv: "7794593386b81652fcd1e0c9ccb37877"
server_key_version: "encryptionkey@0"
I20260620 04:28:54.552856 17767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929496330398-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929496330398-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929496330398-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:28:54.574051 17767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:28:54.576897 17767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:28:54.577066 17767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:28:54.577641 17767 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:28:54.578411 17767 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:28:54.578464 17767 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:28:54.578514 17767 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:28:54.578542 17767 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:28:54.586939 17767 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36113
I20260620 04:28:54.586997 17895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36113 every 8 connection(s)
I20260620 04:28:54.587785 17767 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929496330398-392-0/minicluster-data/ts-2/data/info.pb
I20260620 04:28:54.596226 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 17767
I20260620 04:28:54.596361 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929496330398-392-0/minicluster-data/ts-2/wal/instance
Jun 20 04:28:54 dist-test-slave-nk6d krb5kdc[17392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929734, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:28:54.596709 392 external_mini_cluster.cc:1428] Setting key a4bb89a04312e78b4bd9b4e566db5902
I20260620 04:28:54.600265 17476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:54.589548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49691 (local address 127.0.98.62:36651)
0620 04:28:54.589687 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 04:28:54.589692 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:28:54.590500 (+ 808us) server_negotiation.cc:407] Connection header received
0620 04:28:54.591671 (+ 1171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:28:54.591674 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:28:54.591720 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:28:54.591790 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:28:54.592961 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:28:54.593474 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:28:54.594545 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:28:54.594751 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:28:54.594820 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:28:54.597201 (+ 2381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:28:54.597204 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:28:54.597206 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:28:54.597229 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:28:54.598641 (+ 1412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:28:54.599065 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:28:54.599068 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:28:54.599069 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:28:54.599139 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:28:54.599370 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:28:54.599372 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:28:54.599373 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:28:54.599550 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:28:54.599620 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 04:28:54.600060 (+ 440us) server_negotiation.cc:299] Negotiation successful
0620 04:28:54.600152 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 04:28:54.601037 17898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:54.589913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36651 (local address 127.0.98.3:49691)
0620 04:28:54.590334 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0620 04:28:54.590376 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 04:28:54.591492 (+ 1116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:28:54.591933 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:28:54.591942 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:28:54.592339 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:28:54.592833 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:28:54.592847 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:28:54.593560 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:28:54.593562 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:28:54.594454 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:28:54.594464 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:28:54.594853 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:28:54.594855 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:28:54.594995 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:28:54.595494 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:28:54.595510 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:28:54.597049 (+ 1539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:28:54.598741 (+ 1692us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:28:54.598748 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:28:54.598758 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:28:54.598990 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:28:54.599196 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:28:54.599198 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:28:54.599200 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:28:54.599312 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:28:54.599620 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:28:54.599624 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:28:54.599835 (+ 211us) client_negotiation.cc:769] Sending connection context
0620 04:28:54.600030 (+ 195us) client_negotiation.cc:240] Negotiation successful
0620 04:28:54.600267 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":99,"threads_started":1}
I20260620 04:28:54.601960 17896 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36651
I20260620 04:28:54.602161 17896 heartbeater.cc:464] Registering TS with master...
I20260620 04:28:54.602589 17896 heartbeater.cc:511] Master 127.0.98.62:36651 requested a full tablet report, sending...
I20260620 04:28:54.603485 17437 ts_manager.cc:194] Registered new tserver with Master: 30b82a1886514cb7b07a4d1349d86fcd (127.0.98.3:36113)
I20260620 04:28:54.604048 17437 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:49691
I20260620 04:28:54.610047 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 04:28:54.618880 17476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:54.611823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37438 (local address 127.0.98.62:36651)
0620 04:28:54.611981 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 04:28:54.611985 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:28:54.612144 (+ 159us) server_negotiation.cc:407] Connection header received
0620 04:28:54.612261 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:28:54.612264 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:28:54.612303 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:28:54.612362 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:28:54.612987 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:28:54.613509 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:28:54.614309 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:28:54.614527 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:28:54.614608 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:28:54.615986 (+ 1378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:28:54.615992 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:28:54.615995 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:28:54.616022 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:28:54.617361 (+ 1339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:28:54.617751 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:28:54.617755 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:28:54.617757 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:28:54.617830 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:28:54.618082 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:28:54.618086 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:28:54.618087 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:28:54.618290 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:28:54.618453 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 04:28:54.618633 (+ 180us) server_negotiation.cc:299] Negotiation successful
0620 04:28:54.618734 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 04:28:54.621344 17437 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37438:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 04:28:54.624341 17437 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 04:28:54.637948 17908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:54.632243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34241 (local address 127.0.0.1:53774)
0620 04:28:54.632614 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0620 04:28:54.632666 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 04:28:54.632785 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:28:54.634024 (+ 1239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:28:54.634027 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:28:54.634046 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:28:54.634152 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:28:54.634158 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:28:54.635464 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:28:54.635467 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:28:54.636627 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:28:54.636636 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:28:54.637536 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:28:54.637538 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:28:54.637618 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:28:54.637646 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 04:28:54.637729 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 04:28:54.637809 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"mutex_wait_us":59,"thread_start_us":82,"threads_started":1}
I20260620 04:28:54.638016 17911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:54.633073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53774 (local address 127.0.98.1:34241)
0620 04:28:54.633594 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0620 04:28:54.633606 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 04:28:54.633627 (+ 21us) server_negotiation.cc:407] Connection header received
0620 04:28:54.633696 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:28:54.633701 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:28:54.633883 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:28:54.634041 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:28:54.634258 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:28:54.635325 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:28:54.636731 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:28:54.637438 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:28:54.637542 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:28:54.637681 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 04:28:54.637760 (+ 79us) server_negotiation.cc:299] Negotiation successful
0620 04:28:54.637848 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":87,"threads_started":1}
I20260620 04:28:54.638454 17910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:54.632988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36113 (local address 127.0.0.1:37458)
0620 04:28:54.633249 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0620 04:28:54.633260 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 04:28:54.633351 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:28:54.634297 (+ 946us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:28:54.634300 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:28:54.634315 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:28:54.634404 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:28:54.634410 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:28:54.635316 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:28:54.635319 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:28:54.636889 (+ 1570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:28:54.636900 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:28:54.637911 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:28:54.637915 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:28:54.638004 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:28:54.638026 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 04:28:54.638309 (+ 283us) client_negotiation.cc:240] Negotiation successful
0620 04:28:54.638365 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"mutex_wait_us":114,"thread_start_us":68,"threads_started":1}
I20260620 04:28:54.638870 17912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:54.633487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37458 (local address 127.0.98.3:36113)
0620 04:28:54.633925 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0620 04:28:54.633934 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 04:28:54.633951 (+ 17us) server_negotiation.cc:407] Connection header received
0620 04:28:54.634027 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:28:54.634031 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:28:54.634195 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:28:54.634323 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:28:54.634498 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:28:54.635202 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:28:54.637017 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:28:54.637796 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:28:54.637892 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:28:54.638014 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 04:28:54.638691 (+ 677us) server_negotiation.cc:299] Negotiation successful
0620 04:28:54.638763 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":155,"threads_started":1}
I20260620 04:28:54.639483 17907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:54.632281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46339 (local address 127.0.0.1:42620)
0620 04:28:54.633008 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0620 04:28:54.633021 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:28:54.633103 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:28:54.633802 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:28:54.633808 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:28:54.633830 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:28:54.633969 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:28:54.633977 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:28:54.635403 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:28:54.635407 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:28:54.636889 (+ 1482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:28:54.636900 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:28:54.639093 (+ 2193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:28:54.639096 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:28:54.639197 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:28:54.639220 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 04:28:54.639287 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 04:28:54.639351 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":661,"mutex_wait_us":112,"thread_start_us":75,"threads_started":1}
I20260620 04:28:54.639881 17909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:54.632508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42620 (local address 127.0.98.2:46339)
0620 04:28:54.632921 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0620 04:28:54.632933 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 04:28:54.633115 (+ 182us) server_negotiation.cc:407] Connection header received
0620 04:28:54.633202 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:28:54.633208 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:28:54.633419 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:28:54.633609 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:28:54.634092 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:28:54.635283 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:28:54.638076 (+ 2793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:28:54.638831 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:28:54.638929 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:28:54.639039 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 04:28:54.639670 (+ 631us) server_negotiation.cc:299] Negotiation successful
0620 04:28:54.639765 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":108,"threads_started":1}
I20260620 04:28:54.641270 17830 tablet_service.cc:1467] Processing CreateTablet for tablet 4dfe1959d241444a961421907dcce20c (DEFAULT_TABLE table=test-table [id=7f02f4ab58cd4aa48452edc1103ea37a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:28:54.641144 17562 tablet_service.cc:1467] Processing CreateTablet for tablet 4dfe1959d241444a961421907dcce20c (DEFAULT_TABLE table=test-table [id=7f02f4ab58cd4aa48452edc1103ea37a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:28:54.641842 17696 tablet_service.cc:1467] Processing CreateTablet for tablet 4dfe1959d241444a961421907dcce20c (DEFAULT_TABLE table=test-table [id=7f02f4ab58cd4aa48452edc1103ea37a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:28:54.642210 17562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4dfe1959d241444a961421907dcce20c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:28:54.642210 17830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4dfe1959d241444a961421907dcce20c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:28:54.642588 17696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4dfe1959d241444a961421907dcce20c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:28:54.648415 17913 tablet_bootstrap.cc:492] T 4dfe1959d241444a961421907dcce20c P 4f34bb65e7be435c97df5d178f9d179f: Bootstrap starting.
I20260620 04:28:54.649576 17915 tablet_bootstrap.cc:492] T 4dfe1959d241444a961421907dcce20c P 30b82a1886514cb7b07a4d1349d86fcd: Bootstrap starting.
I20260620 04:28:54.650811 17913 tablet_bootstrap.cc:654] T 4dfe1959d241444a961421907dcce20c P 4f34bb65e7be435c97df5d178f9d179f: Neither blocks nor log segments found. Creating new log.
I20260620 04:28:54.651221 17915 tablet_bootstrap.cc:654] T 4dfe1959d241444a961421907dcce20c P 30b82a1886514cb7b07a4d1349d86fcd: Neither blocks nor log segments found. Creating new log.
I20260620 04:28:54.651779 17913 log.cc:824] T 4dfe1959d241444a961421907dcce20c P 4f34bb65e7be435c97df5d178f9d179f: Log is configured to *not* fsync() on all Append() calls
I20260620 04:28:54.652194 17915 log.cc:824] T 4dfe1959d241444a961421907dcce20c P 30b82a1886514cb7b07a4d1349d86fcd: Log is configured to *not* fsync() on all Append() calls
I20260620 04:28:54.652370 17914 tablet_bootstrap.cc:492] T 4dfe1959d241444a961421907dcce20c P edaf882aed364cb29537e333e87901b3: Bootstrap starting.
I20260620 04:28:54.653568 17913 tablet_bootstrap.cc:492] T 4dfe1959d241444a961421907dcce20c P 4f34bb65e7be435c97df5d178f9d179f: No bootstrap required, opened a new log
I20260620 04:28:54.653740 17913 ts_tablet_manager.cc:1397] T 4dfe1959d241444a961421907dcce20c P 4f34bb65e7be435c97df5d178f9d179f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 04:28:54.653985 17914 tablet_bootstrap.cc:654] T 4dfe1959d241444a961421907dcce20c P edaf882aed364cb29537e333e87901b3: Neither blocks nor log segments found. Creating new log.
I20260620 04:28:54.654844 17914 log.cc:824] T 4dfe1959d241444a961421907dcce20c P edaf882aed364cb29537e333e87901b3: Log is configured to *not* fsync() on all Append() calls
I20260620 04:28:54.656540 17914 tablet_bootstrap.cc:492] T 4dfe1959d241444a961421907dcce20c P edaf882aed364cb29537e333e87901b3: No bootstrap required, opened a new log
I20260620 04:28:54.656721 17914 ts_tablet_manager.cc:1397] T 4dfe1959d241444a961421907dcce20c P edaf882aed364cb29537e333e87901b3: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 04:28:54.657137 17913 raft_consensus.cc:348] T 4dfe1959d241444a961421907dcce20c P 4f34bb65e7be435c97df5d178f9d179f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edaf882aed364cb29537e333e87901b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34241 } } peers { permanent_uuid: "4f34bb65e7be435c97df5d178f9d179f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46339 } } peers { permanent_uuid: "30b82a1886514cb7b07a4d1349d86fcd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36113 } }
I20260620 04:28:54.657383 17913 raft_consensus.cc:374] T 4dfe1959d241444a961421907dcce20c P 4f34bb65e7be435c97df5d178f9d179f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:28:54.657439 17913 raft_consensus.cc:729] T 4dfe1959d241444a961421907dcce20c P 4f34bb65e7be435c97df5d178f9d179f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f34bb65e7be435c97df5d178f9d179f, State: Initialized, Role: FOLLOWER
I20260620 04:28:54.657689 17915 tablet_bootstrap.cc:492] T 4dfe1959d241444a961421907dcce20c P 30b82a1886514cb7b07a4d1349d86fcd: No bootstrap required, opened a new log
I20260620 04:28:54.657860 17915 ts_tablet_manager.cc:1397] T 4dfe1959d241444a961421907dcce20c P 30b82a1886514cb7b07a4d1349d86fcd: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260620 04:28:54.657852 17913 consensus_queue.cc:260] T 4dfe1959d241444a961421907dcce20c P 4f34bb65e7be435c97df5d178f9d179f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edaf882aed364cb29537e333e87901b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34241 } } peers { permanent_uuid: "4f34bb65e7be435c97df5d178f9d179f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46339 } } peers { permanent_uuid: "30b82a1886514cb7b07a4d1349d86fcd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36113 } }
I20260620 04:28:54.658452 17762 heartbeater.cc:503] Master 127.0.98.62:36651 was elected leader, sending a full tablet report...
I20260620 04:28:54.658785 17913 ts_tablet_manager.cc:1428] T 4dfe1959d241444a961421907dcce20c P 4f34bb65e7be435c97df5d178f9d179f: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 04:28:54.660418 17914 raft_consensus.cc:348] T 4dfe1959d241444a961421907dcce20c P edaf882aed364cb29537e333e87901b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edaf882aed364cb29537e333e87901b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34241 } } peers { permanent_uuid: "4f34bb65e7be435c97df5d178f9d179f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46339 } } peers { permanent_uuid: "30b82a1886514cb7b07a4d1349d86fcd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36113 } }
I20260620 04:28:54.660665 17914 raft_consensus.cc:374] T 4dfe1959d241444a961421907dcce20c P edaf882aed364cb29537e333e87901b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:28:54.660718 17914 raft_consensus.cc:729] T 4dfe1959d241444a961421907dcce20c P edaf882aed364cb29537e333e87901b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edaf882aed364cb29537e333e87901b3, State: Initialized, Role: FOLLOWER
I20260620 04:28:54.661151 17914 consensus_queue.cc:260] T 4dfe1959d241444a961421907dcce20c P edaf882aed364cb29537e333e87901b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edaf882aed364cb29537e333e87901b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34241 } } peers { permanent_uuid: "4f34bb65e7be435c97df5d178f9d179f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46339 } } peers { permanent_uuid: "30b82a1886514cb7b07a4d1349d86fcd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36113 } }
I20260620 04:28:54.661474 17915 raft_consensus.cc:348] T 4dfe1959d241444a961421907dcce20c P 30b82a1886514cb7b07a4d1349d86fcd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edaf882aed364cb29537e333e87901b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34241 } } peers { permanent_uuid: "4f34bb65e7be435c97df5d178f9d179f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46339 } } peers { permanent_uuid: "30b82a1886514cb7b07a4d1349d86fcd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36113 } }
I20260620 04:28:54.661710 17915 raft_consensus.cc:374] T 4dfe1959d241444a961421907dcce20c P 30b82a1886514cb7b07a4d1349d86fcd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:28:54.661757 17915 raft_consensus.cc:729] T 4dfe1959d241444a961421907dcce20c P 30b82a1886514cb7b07a4d1349d86fcd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30b82a1886514cb7b07a4d1349d86fcd, State: Initialized, Role: FOLLOWER
I20260620 04:28:54.662110 17914 ts_tablet_manager.cc:1428] T 4dfe1959d241444a961421907dcce20c P edaf882aed364cb29537e333e87901b3: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 04:28:54.662140 17915 consensus_queue.cc:260] T 4dfe1959d241444a961421907dcce20c P 30b82a1886514cb7b07a4d1349d86fcd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edaf882aed364cb29537e333e87901b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34241 } } peers { permanent_uuid: "4f34bb65e7be435c97df5d178f9d179f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46339 } } peers { permanent_uuid: "30b82a1886514cb7b07a4d1349d86fcd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36113 } }
I20260620 04:28:54.662355 17628 heartbeater.cc:503] Master 127.0.98.62:36651 was elected leader, sending a full tablet report...
I20260620 04:28:54.662972 17896 heartbeater.cc:503] Master 127.0.98.62:36651 was elected leader, sending a full tablet report...
I20260620 04:28:54.662994 17915 ts_tablet_manager.cc:1428] T 4dfe1959d241444a961421907dcce20c P 30b82a1886514cb7b07a4d1349d86fcd: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260620 04:28:54.677641 17629 tablet.cc:2369] T 4dfe1959d241444a961421907dcce20c P edaf882aed364cb29537e333e87901b3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 04:28:54.682344 17921 raft_consensus.cc:482] T 4dfe1959d241444a961421907dcce20c P 30b82a1886514cb7b07a4d1349d86fcd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:28:54.682508 17921 raft_consensus.cc:504] T 4dfe1959d241444a961421907dcce20c P 30b82a1886514cb7b07a4d1349d86fcd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edaf882aed364cb29537e333e87901b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34241 } } peers { permanent_uuid: "4f34bb65e7be435c97df5d178f9d179f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46339 } } peers { permanent_uuid: "30b82a1886514cb7b07a4d1349d86fcd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36113 } }
I20260620 04:28:54.683429 17921 leader_election.cc:288] T 4dfe1959d241444a961421907dcce20c P 30b82a1886514cb7b07a4d1349d86fcd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers edaf882aed364cb29537e333e87901b3 (127.0.98.1:34241), 4f34bb65e7be435c97df5d178f9d179f (127.0.98.2:46339)
I20260620 04:28:54.686874 17911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:54.683718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60763 (local address 127.0.98.1:34241)
0620 04:28:54.683844 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 04:28:54.683848 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:28:54.683863 (+ 15us) server_negotiation.cc:407] Connection header received
0620 04:28:54.683986 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:28:54.683988 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:28:54.684037 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:28:54.684120 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:28:54.684365 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:28:54.684941 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:28:54.685951 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:28:54.686364 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:28:54.686449 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:28:54.686495 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 04:28:54.686696 (+ 201us) server_negotiation.cc:299] Negotiation successful
0620 04:28:54.686760 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 04:28:54.686862 17898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:54.683594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34241 (local address 127.0.98.3:60763)
0620 04:28:54.683761 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 04:28:54.683776 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 04:28:54.683853 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:28:54.684131 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:28:54.684133 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:28:54.684151 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:28:54.684271 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:28:54.684278 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:28:54.685040 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:28:54.685043 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:28:54.685831 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:28:54.685840 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:28:54.686474 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:28:54.686478 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:28:54.686579 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:28:54.686604 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 04:28:54.686671 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 04:28:54.686741 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97}
I20260620 04:28:54.686978 17909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:54.683768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34079 (local address 127.0.98.2:46339)
0620 04:28:54.683874 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0620 04:28:54.683878 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:28:54.684129 (+ 251us) server_negotiation.cc:407] Connection header received
0620 04:28:54.684216 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:28:54.684219 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:28:54.684263 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:28:54.684340 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:28:54.684581 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:28:54.685056 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:28:54.686060 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:28:54.686501 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:28:54.686596 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:28:54.686649 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 04:28:54.686816 (+ 167us) server_negotiation.cc:299] Negotiation successful
0620 04:28:54.686878 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 04:28:54.687050 17922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:54.683649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46339 (local address 127.0.98.3:34079)
0620 04:28:54.684047 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0620 04:28:54.684060 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:28:54.684143 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:28:54.684353 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:28:54.684356 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:28:54.684371 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:28:54.684488 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:28:54.684494 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:28:54.685155 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:28:54.685157 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:28:54.685947 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:28:54.685957 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:28:54.686607 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:28:54.686610 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:28:54.686700 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:28:54.686722 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 04:28:54.686791 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 04:28:54.686853 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"mutex_wait_us":1,"thread_start_us":97,"threads_started":1}
I20260620 04:28:54.687534 17582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dfe1959d241444a961421907dcce20c" candidate_uuid: "30b82a1886514cb7b07a4d1349d86fcd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edaf882aed364cb29537e333e87901b3" is_pre_election: true
I20260620 04:28:54.687578 17716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dfe1959d241444a961421907dcce20c" candidate_uuid: "30b82a1886514cb7b07a4d1349d86fcd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f34bb65e7be435c97df5d178f9d179f" is_pre_election: true
I20260620 04:28:54.687810 17716 raft_consensus.cc:2454] T 4dfe1959d241444a961421907dcce20c P 4f34bb65e7be435c97df5d178f9d179f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30b82a1886514cb7b07a4d1349d86fcd in term 0.
I20260620 04:28:54.687811 17582 raft_consensus.cc:2454] T 4dfe1959d241444a961421907dcce20c P edaf882aed364cb29537e333e87901b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30b82a1886514cb7b07a4d1349d86fcd in term 0.
I20260620 04:28:54.688268 17785 leader_election.cc:302] T 4dfe1959d241444a961421907dcce20c P 30b82a1886514cb7b07a4d1349d86fcd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30b82a1886514cb7b07a4d1349d86fcd, 4f34bb65e7be435c97df5d178f9d179f; no voters:
I20260620 04:28:54.688519 17921 raft_consensus.cc:2781] T 4dfe1959d241444a961421907dcce20c P 30b82a1886514cb7b07a4d1349d86fcd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 04:28:54.688587 17921 raft_consensus.cc:482] T 4dfe1959d241444a961421907dcce20c P 30b82a1886514cb7b07a4d1349d86fcd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 04:28:54.688616 17921 raft_consensus.cc:3037] T 4dfe1959d241444a961421907dcce20c P 30b82a1886514cb7b07a4d1349d86fcd [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:28:54.689579 17921 raft_consensus.cc:504] T 4dfe1959d241444a961421907dcce20c P 30b82a1886514cb7b07a4d1349d86fcd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edaf882aed364cb29537e333e87901b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34241 } } peers { permanent_uuid: "4f34bb65e7be435c97df5d178f9d179f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46339 } } peers { permanent_uuid: "30b82a1886514cb7b07a4d1349d86fcd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36113 } }
I20260620 04:28:54.689886 17921 leader_election.cc:288] T 4dfe1959d241444a961421907dcce20c P 30b82a1886514cb7b07a4d1349d86fcd [CANDIDATE]: Term 1 election: Requested vote from peers edaf882aed364cb29537e333e87901b3 (127.0.98.1:34241), 4f34bb65e7be435c97df5d178f9d179f (127.0.98.2:46339)
I20260620 04:28:54.690188 17582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dfe1959d241444a961421907dcce20c" candidate_uuid: "30b82a1886514cb7b07a4d1349d86fcd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edaf882aed364cb29537e333e87901b3"
I20260620 04:28:54.690191 17716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dfe1959d241444a961421907dcce20c" candidate_uuid: "30b82a1886514cb7b07a4d1349d86fcd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f34bb65e7be435c97df5d178f9d179f"
I20260620 04:28:54.690290 17582 raft_consensus.cc:3037] T 4dfe1959d241444a961421907dcce20c P edaf882aed364cb29537e333e87901b3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:28:54.690282 17716 raft_consensus.cc:3037] T 4dfe1959d241444a961421907dcce20c P 4f34bb65e7be435c97df5d178f9d179f [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:28:54.691267 17716 raft_consensus.cc:2454] T 4dfe1959d241444a961421907dcce20c P 4f34bb65e7be435c97df5d178f9d179f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30b82a1886514cb7b07a4d1349d86fcd in term 1.
I20260620 04:28:54.691275 17582 raft_consensus.cc:2454] T 4dfe1959d241444a961421907dcce20c P edaf882aed364cb29537e333e87901b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30b82a1886514cb7b07a4d1349d86fcd in term 1.
I20260620 04:28:54.691525 17786 leader_election.cc:302] T 4dfe1959d241444a961421907dcce20c P 30b82a1886514cb7b07a4d1349d86fcd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30b82a1886514cb7b07a4d1349d86fcd, edaf882aed364cb29537e333e87901b3; no voters:
I20260620 04:28:54.691684 17921 raft_consensus.cc:2781] T 4dfe1959d241444a961421907dcce20c P 30b82a1886514cb7b07a4d1349d86fcd [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:28:54.691890 17921 raft_consensus.cc:686] T 4dfe1959d241444a961421907dcce20c P 30b82a1886514cb7b07a4d1349d86fcd [term 1 LEADER]: Becoming Leader. State: Replica: 30b82a1886514cb7b07a4d1349d86fcd, State: Running, Role: LEADER
I20260620 04:28:54.692247 17921 consensus_queue.cc:237] T 4dfe1959d241444a961421907dcce20c P 30b82a1886514cb7b07a4d1349d86fcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edaf882aed364cb29537e333e87901b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34241 } } peers { permanent_uuid: "4f34bb65e7be435c97df5d178f9d179f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46339 } } peers { permanent_uuid: "30b82a1886514cb7b07a4d1349d86fcd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36113 } }
I20260620 04:28:54.695165 17436 catalog_manager.cc:5495] T 4dfe1959d241444a961421907dcce20c P 30b82a1886514cb7b07a4d1349d86fcd reported cstate change: term changed from 0 to 1, leader changed from <none> to 30b82a1886514cb7b07a4d1349d86fcd (127.0.98.3). New cstate: current_term: 1 leader_uuid: "30b82a1886514cb7b07a4d1349d86fcd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edaf882aed364cb29537e333e87901b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34241 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f34bb65e7be435c97df5d178f9d179f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30b82a1886514cb7b07a4d1349d86fcd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36113 } health_report { overall_health: HEALTHY } } }
I20260620 04:28:54.717656 17912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:54.714351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37462 (local address 127.0.98.3:36113)
0620 04:28:54.714490 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 04:28:54.714493 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:28:54.714504 (+ 11us) server_negotiation.cc:407] Connection header received
0620 04:28:54.714570 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:28:54.714573 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:28:54.714624 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:28:54.714713 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:28:54.714946 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:28:54.715435 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:28:54.716306 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:28:54.716522 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:28:54.716609 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:28:54.716877 (+ 268us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:28:54.717284 (+ 407us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:28:54.717389 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 04:28:54.717447 (+ 58us) server_negotiation.cc:299] Negotiation successful
0620 04:28:54.717510 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 04:28:54.723395 17582 raft_consensus.cc:1261] T 4dfe1959d241444a961421907dcce20c P edaf882aed364cb29537e333e87901b3 [term 1 FOLLOWER]: Refusing update from remote peer 30b82a1886514cb7b07a4d1349d86fcd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:28:54.723395 17716 raft_consensus.cc:1261] T 4dfe1959d241444a961421907dcce20c P 4f34bb65e7be435c97df5d178f9d179f [term 1 FOLLOWER]: Refusing update from remote peer 30b82a1886514cb7b07a4d1349d86fcd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:28:54.724082 17923 consensus_queue.cc:1035] T 4dfe1959d241444a961421907dcce20c P 30b82a1886514cb7b07a4d1349d86fcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f34bb65e7be435c97df5d178f9d179f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:28:54.724244 17921 consensus_queue.cc:1035] T 4dfe1959d241444a961421907dcce20c P 30b82a1886514cb7b07a4d1349d86fcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "edaf882aed364cb29537e333e87901b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:28:54.732281 17928 mvcc.cc:205] Tried to move back new op lower bound from 7298784193419751424 to 7298784193301762048. Current Snapshot: MvccSnapshot[applied={T|T < 7298784193419751424}]
I20260620 04:28:54.739313 17436 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37438:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:28:54.739460 17436 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37438:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:28:54.741667 17436 catalog_manager.cc:5782] T 00000000000000000000000000000000 P eb6f5a8737fe4270867db8d990ae75c3: Sending DeleteTablet for 3 replicas of tablet 4dfe1959d241444a961421907dcce20c
I20260620 04:28:54.742368 17830 tablet_service.cc:1514] Processing DeleteTablet for tablet 4dfe1959d241444a961421907dcce20c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:28:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37458
I20260620 04:28:54.742365 17696 tablet_service.cc:1514] Processing DeleteTablet for tablet 4dfe1959d241444a961421907dcce20c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:28:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42620
I20260620 04:28:54.742365 17562 tablet_service.cc:1514] Processing DeleteTablet for tablet 4dfe1959d241444a961421907dcce20c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:28:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53774
I20260620 04:28:54.742796 17935 tablet_replica.cc:330] T 4dfe1959d241444a961421907dcce20c P 30b82a1886514cb7b07a4d1349d86fcd: stopping tablet replica
I20260620 04:28:54.743115 17935 raft_consensus.cc:2229] T 4dfe1959d241444a961421907dcce20c P 30b82a1886514cb7b07a4d1349d86fcd [term 1 LEADER]: Raft consensus shutting down.
I20260620 04:28:54.743213 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 17499
I20260620 04:28:54.745226 17935 raft_consensus.cc:2258] T 4dfe1959d241444a961421907dcce20c P 30b82a1886514cb7b07a4d1349d86fcd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:28:54.745568 17937 tablet_replica.cc:330] T 4dfe1959d241444a961421907dcce20c P 4f34bb65e7be435c97df5d178f9d179f: stopping tablet replica
I20260620 04:28:54.745867 17937 raft_consensus.cc:2229] T 4dfe1959d241444a961421907dcce20c P 4f34bb65e7be435c97df5d178f9d179f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:28:54.746176 17937 raft_consensus.cc:2258] T 4dfe1959d241444a961421907dcce20c P 4f34bb65e7be435c97df5d178f9d179f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:28:54.746778 17935 ts_tablet_manager.cc:1905] T 4dfe1959d241444a961421907dcce20c P 30b82a1886514cb7b07a4d1349d86fcd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:28:54.749409 17935 ts_tablet_manager.cc:1918] T 4dfe1959d241444a961421907dcce20c P 30b82a1886514cb7b07a4d1349d86fcd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:28:54.749502 17935 log.cc:1196] T 4dfe1959d241444a961421907dcce20c P 30b82a1886514cb7b07a4d1349d86fcd: Deleting WAL directory at /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929496330398-392-0/minicluster-data/ts-2/wal/wals/4dfe1959d241444a961421907dcce20c
I20260620 04:28:54.749850 17935 ts_tablet_manager.cc:1939] T 4dfe1959d241444a961421907dcce20c P 30b82a1886514cb7b07a4d1349d86fcd: Deleting consensus metadata
I20260620 04:28:54.750938 17424 catalog_manager.cc:4841] TS 30b82a1886514cb7b07a4d1349d86fcd (127.0.98.3:36113): tablet 4dfe1959d241444a961421907dcce20c (table test-table [id=7f02f4ab58cd4aa48452edc1103ea37a]) successfully deleted
W20260620 04:28:54.751521 17424 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34241 (error 108)
I20260620 04:28:54.752044 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 17633
I20260620 04:28:54.754495 17907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:54.754205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34241 (local address 127.0.0.1:53776)
0620 04:28:54.754336 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 04:28:54.754411 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 04:28:54.754659 17424 catalog_manager.cc:4584] TS edaf882aed364cb29537e333e87901b3 (127.0.98.1:34241): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4dfe1959d241444a961421907dcce20c: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34241: connect: Connection refused (error 111)
I20260620 04:28:54.758111 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 17767
I20260620 04:28:54.758252 17907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:54.758020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46339 (local address 127.0.0.1:42624)
0620 04:28:54.758127 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0620 04:28:54.758181 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:46339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260620 04:28:54.762830 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 17405
2026-06-20T04:28:54Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2104 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-20T04:28:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:28:54Z Disabled control of system clock
I20260620 04:28:54.789717 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929496330398-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929496330398-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929496330398-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35921
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:43781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35921
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 04:28:54.893173 17942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:28:54.893441 17942 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:28:54.893476 17942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:28:54.897487 17942 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:28:54.897545 17942 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:28:54.897559 17942 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:28:54.897574 17942 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:28:54.897589 17942 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:28:54.902266 17942 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929496330398-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929496330398-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35921
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17942
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929496330398-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:28:54.903280 17942 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:28:54.904651 17942 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:28:54.912364 17947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:28:54.912384 17948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:28:54.912374 17950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:28:54.912649 17942 server_base.cc:1034] running on GCE node
I20260620 04:28:54.913153 17942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:28:54.914227 17942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:28:54.915438 17942 hybrid_clock.cc:648] HybridClock initialized: now 1781929734915391 us; error 24 us; skew 500 ppm
I20260620 04:28:54.917227 17942 webserver.cc:460] Webserver started at http://127.0.98.62:36315/ using document root <none> and password file <none>
I20260620 04:28:54.917765 17942 fs_manager.cc:359] Metadata directory not provided
I20260620 04:28:54.917825 17942 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:28:54.918026 17942 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:28:54.920009 17942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929496330398-392-0/minicluster-data/master-0/data/instance:
uuid: "251eeb534fc94055910206a204e085fb"
format_stamp: "Formatted at 2026-06-20 04:28:54 on dist-test-slave-nk6d"
server_key: "27190bb18a35175841203f048ccdb470"
server_key_iv: "da1eacfb2d7faccacf6db19886b7cd38"
server_key_version: "encryptionkey@0"
I20260620 04:28:54.920548 17942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929496330398-392-0/minicluster-data/master-0/wal/instance:
uuid: "251eeb534fc94055910206a204e085fb"
format_stamp: "Formatted at 2026-06-20 04:28:54 on dist-test-slave-nk6d"
server_key: "27190bb18a35175841203f048ccdb470"
server_key_iv: "da1eacfb2d7faccacf6db19886b7cd38"
server_key_version: "encryptionkey@0"
I20260620 04:28:54.924149 17942 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260620 04:28:54.926512 17956 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:28:54.927760 17942 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:28:54.927872 17942 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929496330398-392-0/minicluster-data/master-0/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929496330398-392-0/minicluster-data/master-0/wal
uuid: "251eeb534fc94055910206a204e085fb"
format_stamp: "Formatted at 2026-06-20 04:28:54 on dist-test-slave-nk6d"
server_key: "27190bb18a35175841203f048ccdb470"
server_key_iv: "da1eacfb2d7faccacf6db19886b7cd38"
server_key_version: "encryptionkey@0"
I20260620 04:28:54.928017 17942 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929496330398-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929496330398-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929496330398-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:28:54.951272 17942 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:28:54.952035 17942 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:28:54.952231 17942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:28:54.958724 17942 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35921
I20260620 04:28:54.958715 18008 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35921 every 8 connection(s)
I20260620 04:28:54.959661 17942 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:28:54.962601 18009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:28:54.964859 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 17942
I20260620 04:28:54.964972 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929496330398-392-0/minicluster-data/master-0/wal/instance
I20260620 04:28:54.965248 392 external_mini_cluster.cc:1428] Setting key 0d33219ba01f3d726b0a152ea6e79e5a
I20260620 04:28:54.968580 18009 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 251eeb534fc94055910206a204e085fb: Bootstrap starting.
I20260620 04:28:54.970679 18009 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 251eeb534fc94055910206a204e085fb: Neither blocks nor log segments found. Creating new log.
I20260620 04:28:54.971522 18009 log.cc:824] T 00000000000000000000000000000000 P 251eeb534fc94055910206a204e085fb: Log is configured to *not* fsync() on all Append() calls
I20260620 04:28:54.973677 18009 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 251eeb534fc94055910206a204e085fb: No bootstrap required, opened a new log
I20260620 04:28:54.974571 18012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:54.966597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49834 (local address 127.0.98.62:35921)
0620 04:28:54.967220 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0620 04:28:54.967235 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 04:28:54.967268 (+ 33us) server_negotiation.cc:407] Connection header received
0620 04:28:54.967923 (+ 655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:28:54.967941 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:28:54.968374 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:28:54.968685 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:28:54.969087 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:28:54.970030 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:28:54.970885 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:28:54.971578 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:28:54.971743 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:28:54.972518 (+ 775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:28:54.972524 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:28:54.972540 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:28:54.972996 (+ 456us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:28:54.973073 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:28:54.973100 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:28:54.973194 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 04:28:54.973357 (+ 163us) server_negotiation.cc:299] Negotiation successful
0620 04:28:54.973670 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":138,"threads_started":1}
I20260620 04:28:54.977192 18009 raft_consensus.cc:348] T 00000000000000000000000000000000 P 251eeb534fc94055910206a204e085fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "251eeb534fc94055910206a204e085fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35921 } }
I20260620 04:28:54.977404 18009 raft_consensus.cc:374] T 00000000000000000000000000000000 P 251eeb534fc94055910206a204e085fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:28:54.977442 18009 raft_consensus.cc:729] T 00000000000000000000000000000000 P 251eeb534fc94055910206a204e085fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 251eeb534fc94055910206a204e085fb, State: Initialized, Role: FOLLOWER
I20260620 04:28:54.977784 18009 consensus_queue.cc:260] T 00000000000000000000000000000000 P 251eeb534fc94055910206a204e085fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "251eeb534fc94055910206a204e085fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35921 } }
I20260620 04:28:54.977890 18009 raft_consensus.cc:388] T 00000000000000000000000000000000 P 251eeb534fc94055910206a204e085fb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:28:54.977931 18009 raft_consensus.cc:482] T 00000000000000000000000000000000 P 251eeb534fc94055910206a204e085fb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:28:54.978003 18009 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 251eeb534fc94055910206a204e085fb [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:28:54.978806 18009 raft_consensus.cc:504] T 00000000000000000000000000000000 P 251eeb534fc94055910206a204e085fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "251eeb534fc94055910206a204e085fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35921 } }
I20260620 04:28:54.979103 18009 leader_election.cc:302] T 00000000000000000000000000000000 P 251eeb534fc94055910206a204e085fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 251eeb534fc94055910206a204e085fb; no voters:
I20260620 04:28:54.979354 18009 leader_election.cc:288] T 00000000000000000000000000000000 P 251eeb534fc94055910206a204e085fb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:28:54.979480 18014 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 251eeb534fc94055910206a204e085fb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:28:54.979687 18014 raft_consensus.cc:686] T 00000000000000000000000000000000 P 251eeb534fc94055910206a204e085fb [term 1 LEADER]: Becoming Leader. State: Replica: 251eeb534fc94055910206a204e085fb, State: Running, Role: LEADER
I20260620 04:28:54.979998 18014 consensus_queue.cc:237] T 00000000000000000000000000000000 P 251eeb534fc94055910206a204e085fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "251eeb534fc94055910206a204e085fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35921 } }
I20260620 04:28:54.980466 18009 sys_catalog.cc:564] T 00000000000000000000000000000000 P 251eeb534fc94055910206a204e085fb [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:28:54.981572 18015 sys_catalog.cc:455] T 00000000000000000000000000000000 P 251eeb534fc94055910206a204e085fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "251eeb534fc94055910206a204e085fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "251eeb534fc94055910206a204e085fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35921 } } }
I20260620 04:28:54.981591 18016 sys_catalog.cc:455] T 00000000000000000000000000000000 P 251eeb534fc94055910206a204e085fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 251eeb534fc94055910206a204e085fb. Latest consensus state: current_term: 1 leader_uuid: "251eeb534fc94055910206a204e085fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "251eeb534fc94055910206a204e085fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35921 } } }
I20260620 04:28:54.981685 18016 sys_catalog.cc:458] T 00000000000000000000000000000000 P 251eeb534fc94055910206a204e085fb [sys.catalog]: This master's current role is: LEADER
I20260620 04:28:54.981683 18015 sys_catalog.cc:458] T 00000000000000000000000000000000 P 251eeb534fc94055910206a204e085fb [sys.catalog]: This master's current role is: LEADER
I20260620 04:28:54.982182 18023 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:28:54.985267 18023 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:28:54.990196 18023 catalog_manager.cc:1347] Generated new cluster ID: f580beb396b14ffcb35235b0a8518810
I20260620 04:28:54.990244 18023 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:28:55.003618 18023 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:28:55.004541 18023 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:28:55.009531 18023 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 251eeb534fc94055910206a204e085fb: Generated new TSK 0
I20260620 04:28:55.010327 18023 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 04:28:55.015096 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929496330398-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929496330398-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929496330398-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929496330398-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35921
--builtin_ntp_servers=127.0.98.20:43781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 04:28:55.122115 18033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:28:55.122365 18033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:28:55.122416 18033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:28:55.126384 18033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:28:55.126441 18033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:28:55.126523 18033 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 04:28:55.131247 18033 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929496330398-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929496330398-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929496330398-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35921
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18033
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929496330398-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:28:55.132335 18033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:28:55.133667 18033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:28:55.142688 18039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:28:55.142804 18033 server_base.cc:1034] running on GCE node
W20260620 04:28:55.142692 18038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:28:55.142706 18041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:28:55.143277 18033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:28:55.143872 18033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:28:55.145025 18033 hybrid_clock.cc:648] HybridClock initialized: now 1781929735144995 us; error 38 us; skew 500 ppm
I20260620 04:28:55.146703 18033 webserver.cc:460] Webserver started at http://127.0.98.1:45043/ using document root <none> and password file <none>
I20260620 04:28:55.147233 18033 fs_manager.cc:359] Metadata directory not provided
I20260620 04:28:55.147295 18033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:28:55.147471 18033 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:28:55.149446 18033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929496330398-392-0/minicluster-data/ts-0/data/instance:
uuid: "ec80a4d41d41456ca924a8d9c356b0f2"
format_stamp: "Formatted at 2026-06-20 04:28:55 on dist-test-slave-nk6d"
server_key: "83e4a85eb00f8d7b0c5c618c08a5da46"
server_key_iv: "dbca8100654b00b0146f1b9276b06367"
server_key_version: "encryptionkey@0"
I20260620 04:28:55.149981 18033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929496330398-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ec80a4d41d41456ca924a8d9c356b0f2"
format_stamp: "Formatted at 2026-06-20 04:28:55 on dist-test-slave-nk6d"
server_key: "83e4a85eb00f8d7b0c5c618c08a5da46"
server_key_iv: "dbca8100654b00b0146f1b9276b06367"
server_key_version: "encryptionkey@0"
I20260620 04:28:55.153718 18033 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.005s
I20260620 04:28:55.156060 18047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:28:55.157189 18033 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20260620 04:28:55.157290 18033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929496330398-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929496330398-392-0/minicluster-data/ts-0/wal
uuid: "ec80a4d41d41456ca924a8d9c356b0f2"
format_stamp: "Formatted at 2026-06-20 04:28:55 on dist-test-slave-nk6d"
server_key: "83e4a85eb00f8d7b0c5c618c08a5da46"
server_key_iv: "dbca8100654b00b0146f1b9276b06367"
server_key_version: "encryptionkey@0"
I20260620 04:28:55.157383 18033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929496330398-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929496330398-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929496330398-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:28:55.176770 18033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:28:55.177418 18033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:28:55.177590 18033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:28:55.178154 18033 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:28:55.179047 18033 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:28:55.179092 18033 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:28:55.179143 18033 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:28:55.179168 18033 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:28:55.187249 18033 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44599
I20260620 04:28:55.187281 18160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44599 every 8 connection(s)
I20260620 04:28:55.188118 18033 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929496330398-392-0/minicluster-data/ts-0/data/info.pb
I20260620 04:28:55.190716 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 18033
I20260620 04:28:55.190825 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929496330398-392-0/minicluster-data/ts-0/wal/instance
I20260620 04:28:55.191123 392 external_mini_cluster.cc:1428] Setting key a9ce82749a25a75126764ba6228ff06c
I20260620 04:28:55.192965 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929496330398-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929496330398-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929496330398-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929496330398-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35921
--builtin_ntp_servers=127.0.98.20:43781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 04:28:55.197172 18012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:55.189821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55005 (local address 127.0.98.62:35921)
0620 04:28:55.189970 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 04:28:55.189974 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:28:55.191068 (+ 1094us) server_negotiation.cc:407] Connection header received
0620 04:28:55.192624 (+ 1556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:28:55.192627 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:28:55.192680 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:28:55.192744 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:28:55.193686 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:28:55.194209 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:28:55.195298 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:28:55.195509 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:28:55.195594 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:28:55.196717 (+ 1123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:28:55.196721 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:28:55.196724 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:28:55.196737 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:28:55.196758 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:28:55.196768 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:28:55.196822 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 04:28:55.196978 (+ 156us) server_negotiation.cc:299] Negotiation successful
0620 04:28:55.197035 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 04:28:55.197973 18163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:55.190226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35921 (local address 127.0.98.1:55005)
0620 04:28:55.190920 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0620 04:28:55.190962 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 04:28:55.192388 (+ 1426us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:28:55.192919 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:28:55.192926 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:28:55.193016 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:28:55.193560 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:28:55.193574 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:28:55.194320 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:28:55.194323 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:28:55.195192 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:28:55.195200 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:28:55.195637 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:28:55.195641 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:28:55.195787 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:28:55.196411 (+ 624us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 04:28:55.196413 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:28:55.196451 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 04:28:55.196529 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:28:55.196844 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:28:55.196853 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:28:55.196858 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 04:28:55.196963 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 04:28:55.197228 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":103,"threads_started":1}
I20260620 04:28:55.199087 18161 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35921
I20260620 04:28:55.199340 18161 heartbeater.cc:464] Registering TS with master...
I20260620 04:28:55.199950 18161 heartbeater.cc:511] Master 127.0.98.62:35921 requested a full tablet report, sending...
I20260620 04:28:55.201587 17973 ts_manager.cc:194] Registered new tserver with Master: ec80a4d41d41456ca924a8d9c356b0f2 (127.0.98.1:44599)
I20260620 04:28:55.202747 17973 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:55005
W20260620 04:28:55.299026 18164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:28:55.299263 18164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:28:55.299300 18164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:28:55.303205 18164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:28:55.303258 18164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:28:55.303313 18164 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 04:28:55.307973 18164 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929496330398-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929496330398-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929496330398-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35921
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18164
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929496330398-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:28:55.308957 18164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:28:55.310317 18164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:28:55.319343 18172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:28:55.319470 18164 server_base.cc:1034] running on GCE node
W20260620 04:28:55.319360 18170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:28:55.319358 18169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:28:55.319950 18164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:28:55.320595 18164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:28:55.321740 18164 hybrid_clock.cc:648] HybridClock initialized: now 1781929735321721 us; error 32 us; skew 500 ppm
I20260620 04:28:55.323442 18164 webserver.cc:460] Webserver started at http://127.0.98.2:35793/ using document root <none> and password file <none>
I20260620 04:28:55.324015 18164 fs_manager.cc:359] Metadata directory not provided
I20260620 04:28:55.324079 18164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:28:55.324254 18164 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:28:55.326186 18164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929496330398-392-0/minicluster-data/ts-1/data/instance:
uuid: "33d7cec25f5b4d30a207d97cb1d13e1e"
format_stamp: "Formatted at 2026-06-20 04:28:55 on dist-test-slave-nk6d"
server_key: "ec44445d3e0775a7c1ba24ddd024cc92"
server_key_iv: "d6feac1457245ad0b56683077665d48b"
server_key_version: "encryptionkey@0"
I20260620 04:28:55.326706 18164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929496330398-392-0/minicluster-data/ts-1/wal/instance:
uuid: "33d7cec25f5b4d30a207d97cb1d13e1e"
format_stamp: "Formatted at 2026-06-20 04:28:55 on dist-test-slave-nk6d"
server_key: "ec44445d3e0775a7c1ba24ddd024cc92"
server_key_iv: "d6feac1457245ad0b56683077665d48b"
server_key_version: "encryptionkey@0"
I20260620 04:28:55.330263 18164 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260620 04:28:55.332639 18178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:28:55.333850 18164 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:28:55.333956 18164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929496330398-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929496330398-392-0/minicluster-data/ts-1/wal
uuid: "33d7cec25f5b4d30a207d97cb1d13e1e"
format_stamp: "Formatted at 2026-06-20 04:28:55 on dist-test-slave-nk6d"
server_key: "ec44445d3e0775a7c1ba24ddd024cc92"
server_key_iv: "d6feac1457245ad0b56683077665d48b"
server_key_version: "encryptionkey@0"
I20260620 04:28:55.334059 18164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929496330398-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929496330398-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929496330398-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:28:55.343386 18164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:28:55.344041 18164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:28:55.344193 18164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:28:55.344738 18164 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:28:55.345644 18164 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:28:55.345688 18164 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:28:55.345728 18164 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:28:55.345757 18164 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:28:55.353927 18164 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45421
I20260620 04:28:55.353945 18291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45421 every 8 connection(s)
I20260620 04:28:55.354827 18164 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929496330398-392-0/minicluster-data/ts-1/data/info.pb
I20260620 04:28:55.358301 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 18164
I20260620 04:28:55.358392 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929496330398-392-0/minicluster-data/ts-1/wal/instance
I20260620 04:28:55.358649 392 external_mini_cluster.cc:1428] Setting key c66e6e77142d5f8deb900ef7fa0ee6b8
I20260620 04:28:55.360631 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929496330398-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929496330398-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929496330398-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929496330398-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35921
--builtin_ntp_servers=127.0.98.20:43781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 04:28:55.363826 18012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:55.356443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52657 (local address 127.0.98.62:35921)
0620 04:28:55.356558 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 04:28:55.356563 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:28:55.357544 (+ 981us) server_negotiation.cc:407] Connection header received
0620 04:28:55.358875 (+ 1331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:28:55.358879 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:28:55.358936 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:28:55.359031 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:28:55.360146 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:28:55.360887 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:28:55.362031 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:28:55.362261 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:28:55.362347 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:28:55.363366 (+ 1019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:28:55.363370 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:28:55.363373 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:28:55.363385 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:28:55.363408 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:28:55.363417 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:28:55.363477 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 04:28:55.363619 (+ 142us) server_negotiation.cc:299] Negotiation successful
0620 04:28:55.363680 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 04:28:55.364760 18294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:55.356793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35921 (local address 127.0.98.2:52657)
0620 04:28:55.357401 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0620 04:28:55.357439 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 04:28:55.358659 (+ 1220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:28:55.359230 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:28:55.359239 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:28:55.359336 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:28:55.360008 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:28:55.360028 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:28:55.361040 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:28:55.361043 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:28:55.361911 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:28:55.361920 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:28:55.362390 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:28:55.362394 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:28:55.362556 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:28:55.363111 (+ 555us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 04:28:55.363112 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:28:55.363152 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 04:28:55.363213 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:28:55.363487 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:28:55.363496 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:28:55.363502 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 04:28:55.363608 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 04:28:55.363889 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":112,"threads_started":1}
I20260620 04:28:55.365772 18292 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35921
I20260620 04:28:55.365981 18292 heartbeater.cc:464] Registering TS with master...
I20260620 04:28:55.366451 18292 heartbeater.cc:511] Master 127.0.98.62:35921 requested a full tablet report, sending...
I20260620 04:28:55.367426 17973 ts_manager.cc:194] Registered new tserver with Master: 33d7cec25f5b4d30a207d97cb1d13e1e (127.0.98.2:45421)
I20260620 04:28:55.368202 17973 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:52657
W20260620 04:28:55.468827 18295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:28:55.469072 18295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:28:55.469108 18295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:28:55.473400 18295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:28:55.473451 18295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:28:55.473508 18295 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 04:28:55.478278 18295 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929496330398-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929496330398-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929496330398-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35921
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18295
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929496330398-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:28:55.479266 18295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:28:55.480650 18295 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:28:55.489730 18300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:28:55.489738 18303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:28:55.489750 18301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:28:55.489886 18295 server_base.cc:1034] running on GCE node
I20260620 04:28:55.490332 18295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:28:55.490916 18295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:28:55.492030 18295 hybrid_clock.cc:648] HybridClock initialized: now 1781929735492012 us; error 33 us; skew 500 ppm
I20260620 04:28:55.493719 18295 webserver.cc:460] Webserver started at http://127.0.98.3:44589/ using document root <none> and password file <none>
I20260620 04:28:55.494251 18295 fs_manager.cc:359] Metadata directory not provided
I20260620 04:28:55.494310 18295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:28:55.494498 18295 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:28:55.496541 18295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929496330398-392-0/minicluster-data/ts-2/data/instance:
uuid: "569105b7ad934527a3d3d41655982d5e"
format_stamp: "Formatted at 2026-06-20 04:28:55 on dist-test-slave-nk6d"
server_key: "204fc532ddbdf51fd05101e2a81ad249"
server_key_iv: "2c6e142d7b48001bee9637447e656b97"
server_key_version: "encryptionkey@0"
I20260620 04:28:55.497062 18295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929496330398-392-0/minicluster-data/ts-2/wal/instance:
uuid: "569105b7ad934527a3d3d41655982d5e"
format_stamp: "Formatted at 2026-06-20 04:28:55 on dist-test-slave-nk6d"
server_key: "204fc532ddbdf51fd05101e2a81ad249"
server_key_iv: "2c6e142d7b48001bee9637447e656b97"
server_key_version: "encryptionkey@0"
I20260620 04:28:55.500537 18295 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 04:28:55.502803 18309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:28:55.504112 18295 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:28:55.504215 18295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929496330398-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929496330398-392-0/minicluster-data/ts-2/wal
uuid: "569105b7ad934527a3d3d41655982d5e"
format_stamp: "Formatted at 2026-06-20 04:28:55 on dist-test-slave-nk6d"
server_key: "204fc532ddbdf51fd05101e2a81ad249"
server_key_iv: "2c6e142d7b48001bee9637447e656b97"
server_key_version: "encryptionkey@0"
I20260620 04:28:55.504307 18295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929496330398-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929496330398-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929496330398-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:28:55.530292 18295 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:28:55.531010 18295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:28:55.531195 18295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:28:55.531811 18295 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:28:55.532773 18295 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:28:55.532820 18295 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:28:55.532876 18295 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:28:55.532903 18295 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:28:55.542016 18295 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34279
I20260620 04:28:55.542066 18422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34279 every 8 connection(s)
I20260620 04:28:55.543174 18295 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929496330398-392-0/minicluster-data/ts-2/data/info.pb
I20260620 04:28:55.546705 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 18295
I20260620 04:28:55.546800 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929496330398-392-0/minicluster-data/ts-2/wal/instance
I20260620 04:28:55.547094 392 external_mini_cluster.cc:1428] Setting key 0a65ef18f797df35fa7b2bc88230f863
I20260620 04:28:55.553074 18012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:55.546829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43821 (local address 127.0.98.62:35921)
0620 04:28:55.546984 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 04:28:55.546990 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 04:28:55.547007 (+ 17us) server_negotiation.cc:407] Connection header received
0620 04:28:55.548509 (+ 1502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:28:55.548512 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:28:55.548573 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:28:55.548624 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:28:55.549814 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:28:55.550330 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:28:55.551443 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:28:55.551693 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:28:55.551767 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:28:55.552720 (+ 953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:28:55.552724 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:28:55.552726 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:28:55.552737 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:28:55.552757 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:28:55.552764 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:28:55.552818 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 04:28:55.552919 (+ 101us) server_negotiation.cc:299] Negotiation successful
0620 04:28:55.552965 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260620 04:28:55.553893 18425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:55.545750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35921 (local address 127.0.98.3:43821)
0620 04:28:55.546644 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0620 04:28:55.546699 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 04:28:55.548283 (+ 1584us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:28:55.549004 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:28:55.549013 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:28:55.549111 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:28:55.549686 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:28:55.549700 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:28:55.550435 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:28:55.550438 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:28:55.551350 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:28:55.551359 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:28:55.551802 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:28:55.551805 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:28:55.551942 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:28:55.552489 (+ 547us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 04:28:55.552490 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:28:55.552530 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 04:28:55.552598 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:28:55.552824 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:28:55.552828 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:28:55.552831 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 04:28:55.552915 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 04:28:55.553124 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":133,"threads_started":1}
I20260620 04:28:55.554855 18423 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35921
I20260620 04:28:55.555063 18423 heartbeater.cc:464] Registering TS with master...
I20260620 04:28:55.555528 18423 heartbeater.cc:511] Master 127.0.98.62:35921 requested a full tablet report, sending...
I20260620 04:28:55.556484 17973 ts_manager.cc:194] Registered new tserver with Master: 569105b7ad934527a3d3d41655982d5e (127.0.98.3:34279)
I20260620 04:28:55.557006 17973 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:43821
I20260620 04:28:55.560914 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 04:28:55.565914 18012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:55.562714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49846 (local address 127.0.98.62:35921)
0620 04:28:55.562840 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 04:28:55.562844 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:28:55.562972 (+ 128us) server_negotiation.cc:407] Connection header received
0620 04:28:55.563080 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:28:55.563082 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:28:55.563119 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:28:55.563177 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:28:55.563379 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:28:55.563868 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:28:55.564646 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:28:55.564846 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:28:55.564918 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:28:55.565597 (+ 679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:28:55.565600 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:28:55.565601 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:28:55.565611 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:28:55.565624 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:28:55.565629 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:28:55.565675 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 04:28:55.565742 (+ 67us) server_negotiation.cc:299] Negotiation successful
0620 04:28:55.565785 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 04:28:55.568287 17973 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49846:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 04:28:55.571194 17973 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 04:28:55.585371 18436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:55.579385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50114 (local address 127.0.98.2:45421)
0620 04:28:55.579755 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0620 04:28:55.579765 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 04:28:55.580420 (+ 655us) server_negotiation.cc:407] Connection header received
0620 04:28:55.580779 (+ 359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:28:55.580785 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:28:55.581009 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:28:55.581169 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:28:55.581672 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:28:55.582826 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:28:55.584201 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:28:55.584735 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:28:55.584834 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:28:55.585031 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0620 04:28:55.585130 (+ 99us) server_negotiation.cc:299] Negotiation successful
0620 04:28:55.585223 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":150,"threads_started":1}
I20260620 04:28:55.585366 18437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:55.579258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45421 (local address 127.0.0.1:50114)
0620 04:28:55.580274 (+ 1016us) negotiation.cc:107] Waiting for socket to connect
0620 04:28:55.580326 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 04:28:55.580583 (+ 257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:28:55.581368 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:28:55.581371 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:28:55.581389 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:28:55.581497 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:28:55.581529 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:28:55.582928 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:28:55.582931 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:28:55.584060 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:28:55.584068 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:28:55.584849 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:28:55.584852 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:28:55.584957 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:28:55.584995 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 04:28:55.585092 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 04:28:55.585213 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":841,"mutex_wait_us":268,"thread_start_us":96,"threads_started":1}
I20260620 04:28:55.586031 18435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:55.579845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34279 (local address 127.0.0.1:46296)
0620 04:28:55.580801 (+ 956us) negotiation.cc:107] Waiting for socket to connect
0620 04:28:55.580815 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:28:55.580908 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:28:55.581512 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:28:55.581514 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:28:55.581540 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:28:55.581687 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:28:55.581696 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:28:55.582779 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:28:55.582783 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:28:55.584057 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:28:55.584068 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:28:55.585398 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:28:55.585403 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:28:55.585671 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:28:55.585699 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 04:28:55.585780 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 04:28:55.585853 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":897,"mutex_wait_us":99,"thread_start_us":117,"threads_started":1}
I20260620 04:28:55.586881 18438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:55.580067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57256 (local address 127.0.98.1:44599)
0620 04:28:55.580762 (+ 695us) server_negotiation.cc:206] Beginning negotiation
0620 04:28:55.580773 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 04:28:55.580796 (+ 23us) server_negotiation.cc:407] Connection header received
0620 04:28:55.580877 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:28:55.580882 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:28:55.581082 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:28:55.581263 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:28:55.581542 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:28:55.582771 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:28:55.584935 (+ 2164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:28:55.585767 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:28:55.585861 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:28:55.586048 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0620 04:28:55.586617 (+ 569us) server_negotiation.cc:299] Negotiation successful
0620 04:28:55.586721 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":135,"threads_started":1}
I20260620 04:28:55.586524 18434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:55.579297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44599 (local address 127.0.0.1:57256)
0620 04:28:55.580639 (+ 1342us) negotiation.cc:107] Waiting for socket to connect
0620 04:28:55.580652 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:28:55.580725 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:28:55.581215 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:28:55.581221 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:28:55.581243 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:28:55.581383 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:28:55.581391 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:28:55.582903 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:28:55.582906 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:28:55.584752 (+ 1846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:28:55.584764 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:28:55.586117 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:28:55.586121 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:28:55.586222 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:28:55.586247 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 04:28:55.586320 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 04:28:55.586389 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1255,"mutex_wait_us":694,"thread_start_us":94,"threads_started":1}
I20260620 04:28:55.588581 18439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:55.580141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46296 (local address 127.0.98.3:34279)
0620 04:28:55.580556 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0620 04:28:55.580568 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 04:28:55.580908 (+ 340us) server_negotiation.cc:407] Connection header received
0620 04:28:55.581010 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:28:55.581015 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:28:55.581179 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:28:55.581361 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:28:55.581803 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:28:55.582643 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:28:55.584201 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:28:55.585022 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:28:55.585128 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:28:55.585331 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0620 04:28:55.588301 (+ 2970us) server_negotiation.cc:299] Negotiation successful
0620 04:28:55.588424 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":128,"threads_started":1}
I20260620 04:28:55.588495 18226 tablet_service.cc:1467] Processing CreateTablet for tablet 0b1adcd145824e7b99c8d4cecefff7e2 (DEFAULT_TABLE table=test-table [id=bea1bea7a8bd42bd8334a4c15a8437b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:28:55.589648 18226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b1adcd145824e7b99c8d4cecefff7e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:28:55.590837 18356 tablet_service.cc:1467] Processing CreateTablet for tablet 0b1adcd145824e7b99c8d4cecefff7e2 (DEFAULT_TABLE table=test-table [id=bea1bea7a8bd42bd8334a4c15a8437b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:28:55.590955 18095 tablet_service.cc:1467] Processing CreateTablet for tablet 0b1adcd145824e7b99c8d4cecefff7e2 (DEFAULT_TABLE table=test-table [id=bea1bea7a8bd42bd8334a4c15a8437b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:28:55.591735 18356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b1adcd145824e7b99c8d4cecefff7e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:28:55.592020 18095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b1adcd145824e7b99c8d4cecefff7e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:28:55.597196 18440 tablet_bootstrap.cc:492] T 0b1adcd145824e7b99c8d4cecefff7e2 P 569105b7ad934527a3d3d41655982d5e: Bootstrap starting.
I20260620 04:28:55.598248 18441 tablet_bootstrap.cc:492] T 0b1adcd145824e7b99c8d4cecefff7e2 P 33d7cec25f5b4d30a207d97cb1d13e1e: Bootstrap starting.
I20260620 04:28:55.598650 18442 tablet_bootstrap.cc:492] T 0b1adcd145824e7b99c8d4cecefff7e2 P ec80a4d41d41456ca924a8d9c356b0f2: Bootstrap starting.
I20260620 04:28:55.599647 18440 tablet_bootstrap.cc:654] T 0b1adcd145824e7b99c8d4cecefff7e2 P 569105b7ad934527a3d3d41655982d5e: Neither blocks nor log segments found. Creating new log.
I20260620 04:28:55.600136 18441 tablet_bootstrap.cc:654] T 0b1adcd145824e7b99c8d4cecefff7e2 P 33d7cec25f5b4d30a207d97cb1d13e1e: Neither blocks nor log segments found. Creating new log.
I20260620 04:28:55.600524 18440 log.cc:824] T 0b1adcd145824e7b99c8d4cecefff7e2 P 569105b7ad934527a3d3d41655982d5e: Log is configured to *not* fsync() on all Append() calls
I20260620 04:28:55.600705 18442 tablet_bootstrap.cc:654] T 0b1adcd145824e7b99c8d4cecefff7e2 P ec80a4d41d41456ca924a8d9c356b0f2: Neither blocks nor log segments found. Creating new log.
I20260620 04:28:55.600896 18441 log.cc:824] T 0b1adcd145824e7b99c8d4cecefff7e2 P 33d7cec25f5b4d30a207d97cb1d13e1e: Log is configured to *not* fsync() on all Append() calls
I20260620 04:28:55.601445 18442 log.cc:824] T 0b1adcd145824e7b99c8d4cecefff7e2 P ec80a4d41d41456ca924a8d9c356b0f2: Log is configured to *not* fsync() on all Append() calls
I20260620 04:28:55.602620 18441 tablet_bootstrap.cc:492] T 0b1adcd145824e7b99c8d4cecefff7e2 P 33d7cec25f5b4d30a207d97cb1d13e1e: No bootstrap required, opened a new log
I20260620 04:28:55.602632 18440 tablet_bootstrap.cc:492] T 0b1adcd145824e7b99c8d4cecefff7e2 P 569105b7ad934527a3d3d41655982d5e: No bootstrap required, opened a new log
I20260620 04:28:55.602797 18440 ts_tablet_manager.cc:1397] T 0b1adcd145824e7b99c8d4cecefff7e2 P 569105b7ad934527a3d3d41655982d5e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 04:28:55.602821 18441 ts_tablet_manager.cc:1397] T 0b1adcd145824e7b99c8d4cecefff7e2 P 33d7cec25f5b4d30a207d97cb1d13e1e: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 04:28:55.603190 18442 tablet_bootstrap.cc:492] T 0b1adcd145824e7b99c8d4cecefff7e2 P ec80a4d41d41456ca924a8d9c356b0f2: No bootstrap required, opened a new log
I20260620 04:28:55.603355 18442 ts_tablet_manager.cc:1397] T 0b1adcd145824e7b99c8d4cecefff7e2 P ec80a4d41d41456ca924a8d9c356b0f2: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 04:28:55.605990 18441 raft_consensus.cc:348] T 0b1adcd145824e7b99c8d4cecefff7e2 P 33d7cec25f5b4d30a207d97cb1d13e1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33d7cec25f5b4d30a207d97cb1d13e1e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45421 } } peers { permanent_uuid: "ec80a4d41d41456ca924a8d9c356b0f2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44599 } } peers { permanent_uuid: "569105b7ad934527a3d3d41655982d5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34279 } }
I20260620 04:28:55.606204 18441 raft_consensus.cc:374] T 0b1adcd145824e7b99c8d4cecefff7e2 P 33d7cec25f5b4d30a207d97cb1d13e1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:28:55.606259 18441 raft_consensus.cc:729] T 0b1adcd145824e7b99c8d4cecefff7e2 P 33d7cec25f5b4d30a207d97cb1d13e1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33d7cec25f5b4d30a207d97cb1d13e1e, State: Initialized, Role: FOLLOWER
I20260620 04:28:55.606719 18441 consensus_queue.cc:260] T 0b1adcd145824e7b99c8d4cecefff7e2 P 33d7cec25f5b4d30a207d97cb1d13e1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33d7cec25f5b4d30a207d97cb1d13e1e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45421 } } peers { permanent_uuid: "ec80a4d41d41456ca924a8d9c356b0f2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44599 } } peers { permanent_uuid: "569105b7ad934527a3d3d41655982d5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34279 } }
I20260620 04:28:55.606859 18440 raft_consensus.cc:348] T 0b1adcd145824e7b99c8d4cecefff7e2 P 569105b7ad934527a3d3d41655982d5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33d7cec25f5b4d30a207d97cb1d13e1e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45421 } } peers { permanent_uuid: "ec80a4d41d41456ca924a8d9c356b0f2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44599 } } peers { permanent_uuid: "569105b7ad934527a3d3d41655982d5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34279 } }
I20260620 04:28:55.607112 18440 raft_consensus.cc:374] T 0b1adcd145824e7b99c8d4cecefff7e2 P 569105b7ad934527a3d3d41655982d5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:28:55.607177 18440 raft_consensus.cc:729] T 0b1adcd145824e7b99c8d4cecefff7e2 P 569105b7ad934527a3d3d41655982d5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 569105b7ad934527a3d3d41655982d5e, State: Initialized, Role: FOLLOWER
I20260620 04:28:55.607434 18292 heartbeater.cc:503] Master 127.0.98.62:35921 was elected leader, sending a full tablet report...
I20260620 04:28:55.607589 18440 consensus_queue.cc:260] T 0b1adcd145824e7b99c8d4cecefff7e2 P 569105b7ad934527a3d3d41655982d5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33d7cec25f5b4d30a207d97cb1d13e1e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45421 } } peers { permanent_uuid: "ec80a4d41d41456ca924a8d9c356b0f2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44599 } } peers { permanent_uuid: "569105b7ad934527a3d3d41655982d5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34279 } }
I20260620 04:28:55.607538 18442 raft_consensus.cc:348] T 0b1adcd145824e7b99c8d4cecefff7e2 P ec80a4d41d41456ca924a8d9c356b0f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33d7cec25f5b4d30a207d97cb1d13e1e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45421 } } peers { permanent_uuid: "ec80a4d41d41456ca924a8d9c356b0f2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44599 } } peers { permanent_uuid: "569105b7ad934527a3d3d41655982d5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34279 } }
I20260620 04:28:55.607756 18442 raft_consensus.cc:374] T 0b1adcd145824e7b99c8d4cecefff7e2 P ec80a4d41d41456ca924a8d9c356b0f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:28:55.607784 18441 ts_tablet_manager.cc:1428] T 0b1adcd145824e7b99c8d4cecefff7e2 P 33d7cec25f5b4d30a207d97cb1d13e1e: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 04:28:55.607800 18442 raft_consensus.cc:729] T 0b1adcd145824e7b99c8d4cecefff7e2 P ec80a4d41d41456ca924a8d9c356b0f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec80a4d41d41456ca924a8d9c356b0f2, State: Initialized, Role: FOLLOWER
I20260620 04:28:55.608260 18442 consensus_queue.cc:260] T 0b1adcd145824e7b99c8d4cecefff7e2 P ec80a4d41d41456ca924a8d9c356b0f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33d7cec25f5b4d30a207d97cb1d13e1e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45421 } } peers { permanent_uuid: "ec80a4d41d41456ca924a8d9c356b0f2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44599 } } peers { permanent_uuid: "569105b7ad934527a3d3d41655982d5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34279 } }
I20260620 04:28:55.608350 18423 heartbeater.cc:503] Master 127.0.98.62:35921 was elected leader, sending a full tablet report...
I20260620 04:28:55.608542 18440 ts_tablet_manager.cc:1428] T 0b1adcd145824e7b99c8d4cecefff7e2 P 569105b7ad934527a3d3d41655982d5e: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 04:28:55.609141 18161 heartbeater.cc:503] Master 127.0.98.62:35921 was elected leader, sending a full tablet report...
I20260620 04:28:55.609966 18442 ts_tablet_manager.cc:1428] T 0b1adcd145824e7b99c8d4cecefff7e2 P ec80a4d41d41456ca924a8d9c356b0f2: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
W20260620 04:28:55.689193 18162 tablet.cc:2369] T 0b1adcd145824e7b99c8d4cecefff7e2 P ec80a4d41d41456ca924a8d9c356b0f2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 04:28:55.791074 18448 raft_consensus.cc:482] T 0b1adcd145824e7b99c8d4cecefff7e2 P ec80a4d41d41456ca924a8d9c356b0f2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:28:55.791275 18448 raft_consensus.cc:504] T 0b1adcd145824e7b99c8d4cecefff7e2 P ec80a4d41d41456ca924a8d9c356b0f2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33d7cec25f5b4d30a207d97cb1d13e1e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45421 } } peers { permanent_uuid: "ec80a4d41d41456ca924a8d9c356b0f2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44599 } } peers { permanent_uuid: "569105b7ad934527a3d3d41655982d5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34279 } }
I20260620 04:28:55.792410 18448 leader_election.cc:288] T 0b1adcd145824e7b99c8d4cecefff7e2 P ec80a4d41d41456ca924a8d9c356b0f2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 33d7cec25f5b4d30a207d97cb1d13e1e (127.0.98.2:45421), 569105b7ad934527a3d3d41655982d5e (127.0.98.3:34279)
W20260620 04:28:55.794265 18424 tablet.cc:2369] T 0b1adcd145824e7b99c8d4cecefff7e2 P 569105b7ad934527a3d3d41655982d5e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 04:28:55.796054 18436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:55.792726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41887 (local address 127.0.98.2:45421)
0620 04:28:55.792846 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 04:28:55.792850 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:28:55.793031 (+ 181us) server_negotiation.cc:407] Connection header received
0620 04:28:55.793158 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:28:55.793162 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:28:55.793207 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:28:55.793268 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:28:55.793537 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:28:55.794034 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:28:55.795156 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:28:55.795526 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:28:55.795614 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:28:55.795673 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 04:28:55.795860 (+ 187us) server_negotiation.cc:299] Negotiation successful
0620 04:28:55.795924 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 04:28:55.796072 18449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:55.792657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45421 (local address 127.0.98.1:41887)
0620 04:28:55.792940 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0620 04:28:55.792953 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:28:55.793053 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:28:55.793288 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:28:55.793290 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:28:55.793305 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:28:55.793434 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:28:55.793439 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:28:55.794131 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:28:55.794135 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:28:55.795023 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:28:55.795032 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:28:55.795634 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:28:55.795638 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:28:55.795733 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:28:55.795773 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 04:28:55.795833 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 04:28:55.795895 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"mutex_wait_us":5,"thread_start_us":69,"threads_started":1}
I20260620 04:28:55.796137 18439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:55.792782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36317 (local address 127.0.98.3:34279)
0620 04:28:55.792901 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 04:28:55.792905 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:28:55.793042 (+ 137us) server_negotiation.cc:407] Connection header received
0620 04:28:55.793163 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:28:55.793165 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:28:55.793210 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:28:55.793272 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:28:55.793546 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:28:55.794027 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:28:55.795215 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:28:55.795618 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:28:55.795705 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:28:55.795759 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 04:28:55.795941 (+ 182us) server_negotiation.cc:299] Negotiation successful
0620 04:28:55.796029 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 04:28:55.796362 18450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:55.792679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34279 (local address 127.0.98.1:36317)
0620 04:28:55.792960 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0620 04:28:55.792970 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 04:28:55.793068 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:28:55.793294 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:28:55.793297 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:28:55.793335 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:28:55.793441 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:28:55.793445 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:28:55.794166 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:28:55.794170 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:28:55.795104 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:28:55.795113 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:28:55.795729 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:28:55.795733 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:28:55.795838 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:28:55.795854 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 04:28:55.795917 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 04:28:55.796022 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"mutex_wait_us":23,"spinlock_wait_cycles":28288,"thread_start_us":72,"threads_started":1}
I20260620 04:28:55.796662 18246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b1adcd145824e7b99c8d4cecefff7e2" candidate_uuid: "ec80a4d41d41456ca924a8d9c356b0f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33d7cec25f5b4d30a207d97cb1d13e1e" is_pre_election: true
I20260620 04:28:55.796864 18369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b1adcd145824e7b99c8d4cecefff7e2" candidate_uuid: "ec80a4d41d41456ca924a8d9c356b0f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "569105b7ad934527a3d3d41655982d5e" is_pre_election: true
I20260620 04:28:55.796928 18246 raft_consensus.cc:2454] T 0b1adcd145824e7b99c8d4cecefff7e2 P 33d7cec25f5b4d30a207d97cb1d13e1e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec80a4d41d41456ca924a8d9c356b0f2 in term 0.
I20260620 04:28:55.797102 18369 raft_consensus.cc:2454] T 0b1adcd145824e7b99c8d4cecefff7e2 P 569105b7ad934527a3d3d41655982d5e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec80a4d41d41456ca924a8d9c356b0f2 in term 0.
I20260620 04:28:55.797420 18051 leader_election.cc:302] T 0b1adcd145824e7b99c8d4cecefff7e2 P ec80a4d41d41456ca924a8d9c356b0f2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33d7cec25f5b4d30a207d97cb1d13e1e, ec80a4d41d41456ca924a8d9c356b0f2; no voters:
I20260620 04:28:55.797649 18448 raft_consensus.cc:2781] T 0b1adcd145824e7b99c8d4cecefff7e2 P ec80a4d41d41456ca924a8d9c356b0f2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 04:28:55.797721 18448 raft_consensus.cc:482] T 0b1adcd145824e7b99c8d4cecefff7e2 P ec80a4d41d41456ca924a8d9c356b0f2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 04:28:55.797751 18448 raft_consensus.cc:3037] T 0b1adcd145824e7b99c8d4cecefff7e2 P ec80a4d41d41456ca924a8d9c356b0f2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:28:55.798715 18448 raft_consensus.cc:504] T 0b1adcd145824e7b99c8d4cecefff7e2 P ec80a4d41d41456ca924a8d9c356b0f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33d7cec25f5b4d30a207d97cb1d13e1e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45421 } } peers { permanent_uuid: "ec80a4d41d41456ca924a8d9c356b0f2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44599 } } peers { permanent_uuid: "569105b7ad934527a3d3d41655982d5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34279 } }
I20260620 04:28:55.799018 18448 leader_election.cc:288] T 0b1adcd145824e7b99c8d4cecefff7e2 P ec80a4d41d41456ca924a8d9c356b0f2 [CANDIDATE]: Term 1 election: Requested vote from peers 33d7cec25f5b4d30a207d97cb1d13e1e (127.0.98.2:45421), 569105b7ad934527a3d3d41655982d5e (127.0.98.3:34279)
I20260620 04:28:55.799330 18246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b1adcd145824e7b99c8d4cecefff7e2" candidate_uuid: "ec80a4d41d41456ca924a8d9c356b0f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33d7cec25f5b4d30a207d97cb1d13e1e"
I20260620 04:28:55.799361 18369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b1adcd145824e7b99c8d4cecefff7e2" candidate_uuid: "ec80a4d41d41456ca924a8d9c356b0f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "569105b7ad934527a3d3d41655982d5e"
I20260620 04:28:55.799435 18246 raft_consensus.cc:3037] T 0b1adcd145824e7b99c8d4cecefff7e2 P 33d7cec25f5b4d30a207d97cb1d13e1e [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:28:55.799458 18369 raft_consensus.cc:3037] T 0b1adcd145824e7b99c8d4cecefff7e2 P 569105b7ad934527a3d3d41655982d5e [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:28:55.800459 18246 raft_consensus.cc:2454] T 0b1adcd145824e7b99c8d4cecefff7e2 P 33d7cec25f5b4d30a207d97cb1d13e1e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec80a4d41d41456ca924a8d9c356b0f2 in term 1.
I20260620 04:28:55.800483 18369 raft_consensus.cc:2454] T 0b1adcd145824e7b99c8d4cecefff7e2 P 569105b7ad934527a3d3d41655982d5e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec80a4d41d41456ca924a8d9c356b0f2 in term 1.
I20260620 04:28:55.800812 18051 leader_election.cc:302] T 0b1adcd145824e7b99c8d4cecefff7e2 P ec80a4d41d41456ca924a8d9c356b0f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33d7cec25f5b4d30a207d97cb1d13e1e, ec80a4d41d41456ca924a8d9c356b0f2; no voters:
I20260620 04:28:55.800994 18448 raft_consensus.cc:2781] T 0b1adcd145824e7b99c8d4cecefff7e2 P ec80a4d41d41456ca924a8d9c356b0f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:28:55.801172 18448 raft_consensus.cc:686] T 0b1adcd145824e7b99c8d4cecefff7e2 P ec80a4d41d41456ca924a8d9c356b0f2 [term 1 LEADER]: Becoming Leader. State: Replica: ec80a4d41d41456ca924a8d9c356b0f2, State: Running, Role: LEADER
I20260620 04:28:55.801487 18448 consensus_queue.cc:237] T 0b1adcd145824e7b99c8d4cecefff7e2 P ec80a4d41d41456ca924a8d9c356b0f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33d7cec25f5b4d30a207d97cb1d13e1e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45421 } } peers { permanent_uuid: "ec80a4d41d41456ca924a8d9c356b0f2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44599 } } peers { permanent_uuid: "569105b7ad934527a3d3d41655982d5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34279 } }
I20260620 04:28:55.804440 17972 catalog_manager.cc:5495] T 0b1adcd145824e7b99c8d4cecefff7e2 P ec80a4d41d41456ca924a8d9c356b0f2 reported cstate change: term changed from 0 to 1, leader changed from <none> to ec80a4d41d41456ca924a8d9c356b0f2 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "ec80a4d41d41456ca924a8d9c356b0f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33d7cec25f5b4d30a207d97cb1d13e1e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec80a4d41d41456ca924a8d9c356b0f2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44599 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "569105b7ad934527a3d3d41655982d5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34279 } health_report { overall_health: UNKNOWN } } }
I20260620 04:28:55.830351 18438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:55.827080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57270 (local address 127.0.98.1:44599)
0620 04:28:55.827206 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 04:28:55.827210 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:28:55.827224 (+ 14us) server_negotiation.cc:407] Connection header received
0620 04:28:55.827355 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:28:55.827358 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:28:55.827412 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:28:55.827497 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:28:55.827674 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:28:55.828240 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:28:55.829092 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:28:55.829308 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:28:55.829387 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:28:55.829621 (+ 234us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:28:55.830024 (+ 403us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:28:55.830138 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 04:28:55.830166 (+ 28us) server_negotiation.cc:299] Negotiation successful
0620 04:28:55.830215 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 04:28:55.836488 18369 raft_consensus.cc:1261] T 0b1adcd145824e7b99c8d4cecefff7e2 P 569105b7ad934527a3d3d41655982d5e [term 1 FOLLOWER]: Refusing update from remote peer ec80a4d41d41456ca924a8d9c356b0f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:28:55.836488 18246 raft_consensus.cc:1261] T 0b1adcd145824e7b99c8d4cecefff7e2 P 33d7cec25f5b4d30a207d97cb1d13e1e [term 1 FOLLOWER]: Refusing update from remote peer ec80a4d41d41456ca924a8d9c356b0f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:28:55.837213 18451 consensus_queue.cc:1035] T 0b1adcd145824e7b99c8d4cecefff7e2 P ec80a4d41d41456ca924a8d9c356b0f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "569105b7ad934527a3d3d41655982d5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:28:55.837410 18448 consensus_queue.cc:1035] T 0b1adcd145824e7b99c8d4cecefff7e2 P ec80a4d41d41456ca924a8d9c356b0f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33d7cec25f5b4d30a207d97cb1d13e1e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:28:55.846483 18455 mvcc.cc:205] Tried to move back new op lower bound from 7298784197978447872 to 7298784197845200896. Current Snapshot: MvccSnapshot[applied={T|T < 7298784197978447872}]
I20260620 04:28:55.853139 17972 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49846:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:28:55.853304 17972 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49846:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:28:55.855545 17972 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 251eeb534fc94055910206a204e085fb: Sending DeleteTablet for 3 replicas of tablet 0b1adcd145824e7b99c8d4cecefff7e2
I20260620 04:28:55.856338 18226 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b1adcd145824e7b99c8d4cecefff7e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:28:55 UTC) from {username='slave'} at 127.0.0.1:50114
I20260620 04:28:55.856599 18095 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b1adcd145824e7b99c8d4cecefff7e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:28:55 UTC) from {username='slave'} at 127.0.0.1:57256
I20260620 04:28:55.856873 18463 tablet_replica.cc:330] T 0b1adcd145824e7b99c8d4cecefff7e2 P 33d7cec25f5b4d30a207d97cb1d13e1e: stopping tablet replica
I20260620 04:28:55.857049 18464 tablet_replica.cc:330] T 0b1adcd145824e7b99c8d4cecefff7e2 P ec80a4d41d41456ca924a8d9c356b0f2: stopping tablet replica
I20260620 04:28:55.857206 18463 raft_consensus.cc:2229] T 0b1adcd145824e7b99c8d4cecefff7e2 P 33d7cec25f5b4d30a207d97cb1d13e1e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:28:55.857357 18464 raft_consensus.cc:2229] T 0b1adcd145824e7b99c8d4cecefff7e2 P ec80a4d41d41456ca924a8d9c356b0f2 [term 1 LEADER]: Raft consensus shutting down.
I20260620 04:28:55.857527 18463 raft_consensus.cc:2258] T 0b1adcd145824e7b99c8d4cecefff7e2 P 33d7cec25f5b4d30a207d97cb1d13e1e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:28:55.857518 18356 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b1adcd145824e7b99c8d4cecefff7e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:28:55 UTC) from {username='slave'} at 127.0.0.1:46296
I20260620 04:28:55.857857 18464 raft_consensus.cc:2258] T 0b1adcd145824e7b99c8d4cecefff7e2 P ec80a4d41d41456ca924a8d9c356b0f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:28:55.858670 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 18033
I20260620 04:28:55.858973 18463 ts_tablet_manager.cc:1905] T 0b1adcd145824e7b99c8d4cecefff7e2 P 33d7cec25f5b4d30a207d97cb1d13e1e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:28:55.859269 18465 tablet_replica.cc:330] T 0b1adcd145824e7b99c8d4cecefff7e2 P 569105b7ad934527a3d3d41655982d5e: stopping tablet replica
I20260620 04:28:55.859539 18465 raft_consensus.cc:2229] T 0b1adcd145824e7b99c8d4cecefff7e2 P 569105b7ad934527a3d3d41655982d5e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:28:55.859787 18465 raft_consensus.cc:2258] T 0b1adcd145824e7b99c8d4cecefff7e2 P 569105b7ad934527a3d3d41655982d5e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:28:55.861155 18465 ts_tablet_manager.cc:1905] T 0b1adcd145824e7b99c8d4cecefff7e2 P 569105b7ad934527a3d3d41655982d5e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:28:55.861644 18463 ts_tablet_manager.cc:1918] T 0b1adcd145824e7b99c8d4cecefff7e2 P 33d7cec25f5b4d30a207d97cb1d13e1e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:28:55.861732 18463 log.cc:1196] T 0b1adcd145824e7b99c8d4cecefff7e2 P 33d7cec25f5b4d30a207d97cb1d13e1e: Deleting WAL directory at /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929496330398-392-0/minicluster-data/ts-1/wal/wals/0b1adcd145824e7b99c8d4cecefff7e2
I20260620 04:28:55.862072 18463 ts_tablet_manager.cc:1939] T 0b1adcd145824e7b99c8d4cecefff7e2 P 33d7cec25f5b4d30a207d97cb1d13e1e: Deleting consensus metadata
I20260620 04:28:55.863227 17960 catalog_manager.cc:4841] TS 33d7cec25f5b4d30a207d97cb1d13e1e (127.0.98.2:45421): tablet 0b1adcd145824e7b99c8d4cecefff7e2 (table test-table [id=bea1bea7a8bd42bd8334a4c15a8437b0]) successfully deleted
I20260620 04:28:55.863492 18465 ts_tablet_manager.cc:1918] T 0b1adcd145824e7b99c8d4cecefff7e2 P 569105b7ad934527a3d3d41655982d5e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:28:55.863577 18465 log.cc:1196] T 0b1adcd145824e7b99c8d4cecefff7e2 P 569105b7ad934527a3d3d41655982d5e: Deleting WAL directory at /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929496330398-392-0/minicluster-data/ts-2/wal/wals/0b1adcd145824e7b99c8d4cecefff7e2
I20260620 04:28:55.863857 18465 ts_tablet_manager.cc:1939] T 0b1adcd145824e7b99c8d4cecefff7e2 P 569105b7ad934527a3d3d41655982d5e: Deleting consensus metadata
I20260620 04:28:55.864885 17958 catalog_manager.cc:4841] TS 569105b7ad934527a3d3d41655982d5e (127.0.98.3:34279): tablet 0b1adcd145824e7b99c8d4cecefff7e2 (table test-table [id=bea1bea7a8bd42bd8334a4c15a8437b0]) successfully deleted
W20260620 04:28:55.865721 17957 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44599 (error 108)
I20260620 04:28:55.865926 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 18164
I20260620 04:28:55.870817 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 18295
I20260620 04:28:55.871088 18434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:55.870774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44599 (local address 127.0.0.1:57274)
0620 04:28:55.870904 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 04:28:55.870974 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260620 04:28:55.871443 17957 catalog_manager.cc:4584] TS ec80a4d41d41456ca924a8d9c356b0f2 (127.0.98.1:44599): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0b1adcd145824e7b99c8d4cecefff7e2: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44599: connect: Connection refused (error 111)
I20260620 04:28:55.875120 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 17942
2026-06-20T04:28:55Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1111 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929496330398-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:28:55 dist-test-slave-nk6d krb5kdc[18469](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:28:55 dist-test-slave-nk6d krb5kdc[18469](info): set up 2 sockets
Jun 20 04:28:55 dist-test-slave-nk6d krb5kdc[18469](info): commencing operation
krb5kdc: starting...
W20260620 04:28:56.823648 392 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929496330398-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929496330398-392-0/krb5kdc/test-user.keytab.
Jun 20 04:28:56 dist-test-slave-nk6d krb5kdc[18469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929736, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:28:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:28:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 04:28:56.934906 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929496330398-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929496330398-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929496330398-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34197
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34197
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:28:57.040323 18482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:28:57.040577 18482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:28:57.040633 18482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:28:57.044673 18482 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:28:57.044739 18482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:28:57.044766 18482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:28:57.044786 18482 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:28:57.044807 18482 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:28:57.050140 18482 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929496330398-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929496330398-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34197
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34197
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18482
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929496330398-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:28:57.051164 18482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:28:57.052176 18482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:28:57.060122 18488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:28:57.060124 18490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:28:57.060133 18487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:28:57.060458 18482 server_base.cc:1034] running on GCE node
I20260620 04:28:57.060994 18482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:28:57.062076 18482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:28:57.063238 18482 hybrid_clock.cc:648] HybridClock initialized: now 1781929737063221 us; error 31 us; skew 500 ppm
Jun 20 04:28:57 dist-test-slave-nk6d krb5kdc[18469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929737, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:28:57.066010 18482 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 04:28:57.066859 18482 webserver.cc:460] Webserver started at http://127.0.98.62:39023/ using document root <none> and password file <none>
I20260620 04:28:57.067384 18482 fs_manager.cc:359] Metadata directory not provided
I20260620 04:28:57.067446 18482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:28:57.067636 18482 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:28:57.069682 18482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929496330398-392-0/minicluster-data/master-0/data/instance:
uuid: "de4c89a3d70a491ba1bce513a4a7f6de"
format_stamp: "Formatted at 2026-06-20 04:28:57 on dist-test-slave-nk6d"
server_key: "e0797aaf1676b26073607455193bd65b"
server_key_iv: "f630c24f9dfde035bed5dc85a2235f39"
server_key_version: "encryptionkey@0"
I20260620 04:28:57.070220 18482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929496330398-392-0/minicluster-data/master-0/wal/instance:
uuid: "de4c89a3d70a491ba1bce513a4a7f6de"
format_stamp: "Formatted at 2026-06-20 04:28:57 on dist-test-slave-nk6d"
server_key: "e0797aaf1676b26073607455193bd65b"
server_key_iv: "f630c24f9dfde035bed5dc85a2235f39"
server_key_version: "encryptionkey@0"
I20260620 04:28:57.073737 18482 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 04:28:57.076097 18497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:28:57.077292 18482 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:28:57.077400 18482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929496330398-392-0/minicluster-data/master-0/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929496330398-392-0/minicluster-data/master-0/wal
uuid: "de4c89a3d70a491ba1bce513a4a7f6de"
format_stamp: "Formatted at 2026-06-20 04:28:57 on dist-test-slave-nk6d"
server_key: "e0797aaf1676b26073607455193bd65b"
server_key_iv: "f630c24f9dfde035bed5dc85a2235f39"
server_key_version: "encryptionkey@0"
I20260620 04:28:57.077494 18482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929496330398-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929496330398-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929496330398-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:28:57.094434 18482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:28:57.097304 18482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:28:57.097476 18482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:28:57.104264 18482 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34197
I20260620 04:28:57.104287 18549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34197 every 8 connection(s)
I20260620 04:28:57.105196 18482 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:28:57.107903 18550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:28:57.110607 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 18482
I20260620 04:28:57.110718 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929496330398-392-0/minicluster-data/master-0/wal/instance
I20260620 04:28:57.110971 392 external_mini_cluster.cc:1428] Setting key ca5350853c5c984a594a5e7f3311fc71
I20260620 04:28:57.113660 18550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de4c89a3d70a491ba1bce513a4a7f6de: Bootstrap starting.
I20260620 04:28:57.115726 18550 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P de4c89a3d70a491ba1bce513a4a7f6de: Neither blocks nor log segments found. Creating new log.
I20260620 04:28:57.116604 18550 log.cc:824] T 00000000000000000000000000000000 P de4c89a3d70a491ba1bce513a4a7f6de: Log is configured to *not* fsync() on all Append() calls
I20260620 04:28:57.118813 18550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de4c89a3d70a491ba1bce513a4a7f6de: No bootstrap required, opened a new log
Jun 20 04:28:57 dist-test-slave-nk6d krb5kdc[18469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929736, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:28:57.122164 18550 raft_consensus.cc:348] T 00000000000000000000000000000000 P de4c89a3d70a491ba1bce513a4a7f6de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de4c89a3d70a491ba1bce513a4a7f6de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34197 } }
I20260620 04:28:57.122385 18550 raft_consensus.cc:374] T 00000000000000000000000000000000 P de4c89a3d70a491ba1bce513a4a7f6de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:28:57.122444 18550 raft_consensus.cc:729] T 00000000000000000000000000000000 P de4c89a3d70a491ba1bce513a4a7f6de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de4c89a3d70a491ba1bce513a4a7f6de, State: Initialized, Role: FOLLOWER
I20260620 04:28:57.122848 18550 consensus_queue.cc:260] T 00000000000000000000000000000000 P de4c89a3d70a491ba1bce513a4a7f6de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de4c89a3d70a491ba1bce513a4a7f6de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34197 } }
I20260620 04:28:57.123003 18550 raft_consensus.cc:388] T 00000000000000000000000000000000 P de4c89a3d70a491ba1bce513a4a7f6de [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:28:57.123066 18550 raft_consensus.cc:482] T 00000000000000000000000000000000 P de4c89a3d70a491ba1bce513a4a7f6de [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:28:57.123162 18550 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de4c89a3d70a491ba1bce513a4a7f6de [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:28:57.124100 18550 raft_consensus.cc:504] T 00000000000000000000000000000000 P de4c89a3d70a491ba1bce513a4a7f6de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de4c89a3d70a491ba1bce513a4a7f6de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34197 } }
I20260620 04:28:57.124455 18550 leader_election.cc:302] T 00000000000000000000000000000000 P de4c89a3d70a491ba1bce513a4a7f6de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: de4c89a3d70a491ba1bce513a4a7f6de; no voters:
I20260620 04:28:57.124792 18550 leader_election.cc:288] T 00000000000000000000000000000000 P de4c89a3d70a491ba1bce513a4a7f6de [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:28:57.125119 18556 raft_consensus.cc:2781] T 00000000000000000000000000000000 P de4c89a3d70a491ba1bce513a4a7f6de [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:28:57.125324 18556 raft_consensus.cc:686] T 00000000000000000000000000000000 P de4c89a3d70a491ba1bce513a4a7f6de [term 1 LEADER]: Becoming Leader. State: Replica: de4c89a3d70a491ba1bce513a4a7f6de, State: Running, Role: LEADER
I20260620 04:28:57.125684 18556 consensus_queue.cc:237] T 00000000000000000000000000000000 P de4c89a3d70a491ba1bce513a4a7f6de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de4c89a3d70a491ba1bce513a4a7f6de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34197 } }
I20260620 04:28:57.125861 18553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:57.112346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53174 (local address 127.0.98.62:34197)
0620 04:28:57.112789 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0620 04:28:57.112804 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 04:28:57.112840 (+ 36us) server_negotiation.cc:407] Connection header received
0620 04:28:57.113421 (+ 581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:28:57.113437 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:28:57.113755 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:28:57.114076 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:28:57.114846 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:28:57.115820 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:28:57.117009 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:28:57.117607 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:28:57.117724 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:28:57.120495 (+ 2771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:28:57.120500 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:28:57.120512 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:28:57.120538 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:28:57.122806 (+ 2268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:28:57.123269 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:28:57.123274 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:28:57.123280 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:28:57.123409 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:28:57.123690 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:28:57.123692 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:28:57.123693 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:28:57.124131 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:28:57.124318 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0620 04:28:57.124563 (+ 245us) server_negotiation.cc:299] Negotiation successful
0620 04:28:57.124864 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":138,"threads_started":1}
I20260620 04:28:57.125942 18550 sys_catalog.cc:564] T 00000000000000000000000000000000 P de4c89a3d70a491ba1bce513a4a7f6de [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:28:57.126852 18556 sys_catalog.cc:455] T 00000000000000000000000000000000 P de4c89a3d70a491ba1bce513a4a7f6de [sys.catalog]: SysCatalogTable state changed. Reason: New leader de4c89a3d70a491ba1bce513a4a7f6de. Latest consensus state: current_term: 1 leader_uuid: "de4c89a3d70a491ba1bce513a4a7f6de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de4c89a3d70a491ba1bce513a4a7f6de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34197 } } }
I20260620 04:28:57.127100 18556 sys_catalog.cc:458] T 00000000000000000000000000000000 P de4c89a3d70a491ba1bce513a4a7f6de [sys.catalog]: This master's current role is: LEADER
I20260620 04:28:57.127532 18557 sys_catalog.cc:455] T 00000000000000000000000000000000 P de4c89a3d70a491ba1bce513a4a7f6de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "de4c89a3d70a491ba1bce513a4a7f6de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de4c89a3d70a491ba1bce513a4a7f6de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34197 } } }
I20260620 04:28:57.127614 18557 sys_catalog.cc:458] T 00000000000000000000000000000000 P de4c89a3d70a491ba1bce513a4a7f6de [sys.catalog]: This master's current role is: LEADER
I20260620 04:28:57.128050 18564 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:28:57.131181 18564 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:28:57.136426 18564 catalog_manager.cc:1347] Generated new cluster ID: aa25c640b44c4685b975865b15a70dba
I20260620 04:28:57.136487 18564 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:28:57.163232 18564 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:28:57.164646 18564 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:28:57.177985 18564 catalog_manager.cc:5868] T 00000000000000000000000000000000 P de4c89a3d70a491ba1bce513a4a7f6de: Generated new TSK 0
I20260620 04:28:57.178709 18564 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929496330398-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929496330398-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 04:28:57.212291 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929496330398-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929496330398-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929496330398-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929496330398-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34197
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929496330398-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:28:57.319806 18576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:28:57.320099 18576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:28:57.320142 18576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:28:57.324050 18576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:28:57.324112 18576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:28:57.324177 18576 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 04:28:57.329481 18576 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929496330398-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929496330398-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929496330398-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929496330398-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34197
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18576
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929496330398-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:28:57.330490 18576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:28:57.331459 18576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:28:57.340615 18581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:28:57.340663 18576 server_base.cc:1034] running on GCE node
W20260620 04:28:57.340627 18582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:28:57.340639 18584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:28:57.341104 18576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:28:57.341698 18576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:28:57.342850 18576 hybrid_clock.cc:648] HybridClock initialized: now 1781929737342833 us; error 30 us; skew 500 ppm
Jun 20 04:28:57 dist-test-slave-nk6d krb5kdc[18469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929737, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:28:57.345448 18576 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 04:28:57.346221 18576 webserver.cc:460] Webserver started at http://127.0.98.1:41671/ using document root <none> and password file <none>
I20260620 04:28:57.346736 18576 fs_manager.cc:359] Metadata directory not provided
I20260620 04:28:57.346799 18576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:28:57.346975 18576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:28:57.348997 18576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929496330398-392-0/minicluster-data/ts-0/data/instance:
uuid: "e62476cda7b94339abcd5c3df544cfdd"
format_stamp: "Formatted at 2026-06-20 04:28:57 on dist-test-slave-nk6d"
server_key: "03c4319c5f7898222d33f69ecc6416e7"
server_key_iv: "21c53811205e63119940f48dadca155c"
server_key_version: "encryptionkey@0"
I20260620 04:28:57.349501 18576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929496330398-392-0/minicluster-data/ts-0/wal/instance:
uuid: "e62476cda7b94339abcd5c3df544cfdd"
format_stamp: "Formatted at 2026-06-20 04:28:57 on dist-test-slave-nk6d"
server_key: "03c4319c5f7898222d33f69ecc6416e7"
server_key_iv: "21c53811205e63119940f48dadca155c"
server_key_version: "encryptionkey@0"
I20260620 04:28:57.354441 18576 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 04:28:57.357446 18591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:28:57.358764 18576 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 04:28:57.358863 18576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929496330398-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929496330398-392-0/minicluster-data/ts-0/wal
uuid: "e62476cda7b94339abcd5c3df544cfdd"
format_stamp: "Formatted at 2026-06-20 04:28:57 on dist-test-slave-nk6d"
server_key: "03c4319c5f7898222d33f69ecc6416e7"
server_key_iv: "21c53811205e63119940f48dadca155c"
server_key_version: "encryptionkey@0"
I20260620 04:28:57.358973 18576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929496330398-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929496330398-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929496330398-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:28:57.372262 18576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:28:57.374888 18576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:28:57.375038 18576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:28:57.375561 18576 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:28:57.376362 18576 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:28:57.376413 18576 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:28:57.376454 18576 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:28:57.376482 18576 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:28:57.384589 18576 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36563
I20260620 04:28:57.384635 18704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36563 every 8 connection(s)
I20260620 04:28:57.385425 18576 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929496330398-392-0/minicluster-data/ts-0/data/info.pb
I20260620 04:28:57.388123 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 18576
I20260620 04:28:57.388245 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929496330398-392-0/minicluster-data/ts-0/wal/instance
I20260620 04:28:57.388563 392 external_mini_cluster.cc:1428] Setting key 29ee1bb67552b2080719dcb4e64e3ccd
Jun 20 04:28:57 dist-test-slave-nk6d krb5kdc[18469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929737, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 04:28:57.398803 18553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:57.387046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44103 (local address 127.0.98.62:34197)
0620 04:28:57.387165 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 04:28:57.387169 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:28:57.387979 (+ 810us) server_negotiation.cc:407] Connection header received
0620 04:28:57.389512 (+ 1533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:28:57.389515 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:28:57.389555 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:28:57.389622 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:28:57.391007 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:28:57.391532 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:28:57.392702 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:28:57.392941 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:28:57.393028 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:28:57.395326 (+ 2298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:28:57.395330 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:28:57.395331 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:28:57.395355 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:28:57.396816 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:28:57.397304 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:28:57.397307 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:28:57.397309 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:28:57.397387 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:28:57.397716 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:28:57.397719 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:28:57.397721 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:28:57.397910 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:28:57.397995 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 04:28:57.398495 (+ 500us) server_negotiation.cc:299] Negotiation successful
0620 04:28:57.398625 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 04:28:57.399864 18707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:57.387393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34197 (local address 127.0.98.1:44103)
0620 04:28:57.387805 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0620 04:28:57.387857 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 04:28:57.389298 (+ 1441us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:28:57.389792 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:28:57.389798 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:28:57.390222 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:28:57.390858 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:28:57.390876 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:28:57.391647 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:28:57.391650 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:28:57.392582 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:28:57.392592 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:28:57.393074 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:28:57.393078 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:28:57.393231 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:28:57.393756 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:28:57.393771 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:28:57.395171 (+ 1400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:28:57.396936 (+ 1765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:28:57.396943 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:28:57.396957 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:28:57.397206 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:28:57.397469 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:28:57.397472 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:28:57.397474 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:28:57.397606 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:28:57.398007 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:28:57.398015 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:28:57.398259 (+ 244us) client_negotiation.cc:769] Sending connection context
0620 04:28:57.398470 (+ 211us) client_negotiation.cc:240] Negotiation successful
0620 04:28:57.398850 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":110,"threads_started":1}
I20260620 04:28:57.401077 18705 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34197
I20260620 04:28:57.401324 18705 heartbeater.cc:464] Registering TS with master...
I20260620 04:28:57.401794 18705 heartbeater.cc:511] Master 127.0.98.62:34197 requested a full tablet report, sending...
I20260620 04:28:57.403208 18509 ts_manager.cc:194] Registered new tserver with Master: e62476cda7b94339abcd5c3df544cfdd (127.0.98.1:36563)
I20260620 04:28:57.404458 18509 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:44103
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929496330398-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929496330398-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 04:28:57.413039 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929496330398-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929496330398-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929496330398-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929496330398-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34197
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929496330398-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:28:57.519272 18710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:28:57.519521 18710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:28:57.519562 18710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:28:57.523415 18710 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:28:57.523469 18710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:28:57.523525 18710 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 04:28:57.528649 18710 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929496330398-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929496330398-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929496330398-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929496330398-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34197
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18710
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929496330398-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:28:57.529628 18710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:28:57.530587 18710 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:28:57.539666 18715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:28:57.539685 18716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:28:57.539705 18718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:28:57.539758 18710 server_base.cc:1034] running on GCE node
I20260620 04:28:57.540298 18710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:28:57.540856 18710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:28:57.542049 18710 hybrid_clock.cc:648] HybridClock initialized: now 1781929737542021 us; error 40 us; skew 500 ppm
Jun 20 04:28:57 dist-test-slave-nk6d krb5kdc[18469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929737, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:28:57.545316 18710 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 04:28:57.546382 18710 webserver.cc:460] Webserver started at http://127.0.98.2:42203/ using document root <none> and password file <none>
I20260620 04:28:57.547037 18710 fs_manager.cc:359] Metadata directory not provided
I20260620 04:28:57.547099 18710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:28:57.547256 18710 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:28:57.549218 18710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929496330398-392-0/minicluster-data/ts-1/data/instance:
uuid: "daca0dbd55b64835865f0a771657e390"
format_stamp: "Formatted at 2026-06-20 04:28:57 on dist-test-slave-nk6d"
server_key: "2fa2ae04e4ce1d8a2c3ea95fc2d3179f"
server_key_iv: "bcfd63cc0605184a63612f0ea689a6b9"
server_key_version: "encryptionkey@0"
I20260620 04:28:57.549700 18710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929496330398-392-0/minicluster-data/ts-1/wal/instance:
uuid: "daca0dbd55b64835865f0a771657e390"
format_stamp: "Formatted at 2026-06-20 04:28:57 on dist-test-slave-nk6d"
server_key: "2fa2ae04e4ce1d8a2c3ea95fc2d3179f"
server_key_iv: "bcfd63cc0605184a63612f0ea689a6b9"
server_key_version: "encryptionkey@0"
I20260620 04:28:57.553166 18710 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260620 04:28:57.555352 18725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:28:57.556543 18710 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:28:57.556639 18710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929496330398-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929496330398-392-0/minicluster-data/ts-1/wal
uuid: "daca0dbd55b64835865f0a771657e390"
format_stamp: "Formatted at 2026-06-20 04:28:57 on dist-test-slave-nk6d"
server_key: "2fa2ae04e4ce1d8a2c3ea95fc2d3179f"
server_key_iv: "bcfd63cc0605184a63612f0ea689a6b9"
server_key_version: "encryptionkey@0"
I20260620 04:28:57.556717 18710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929496330398-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929496330398-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929496330398-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:28:57.581671 18710 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:28:57.584767 18710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:28:57.584924 18710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:28:57.585448 18710 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:28:57.586213 18710 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:28:57.586262 18710 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:28:57.586309 18710 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:28:57.586328 18710 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:28:57.594609 18710 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40669
I20260620 04:28:57.594646 18838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40669 every 8 connection(s)
I20260620 04:28:57.595453 18710 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929496330398-392-0/minicluster-data/ts-1/data/info.pb
I20260620 04:28:57.598577 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 18710
I20260620 04:28:57.598727 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929496330398-392-0/minicluster-data/ts-1/wal/instance
I20260620 04:28:57.599036 392 external_mini_cluster.cc:1428] Setting key 0588842ecee437a006148375e8f93db5
Jun 20 04:28:57 dist-test-slave-nk6d krb5kdc[18469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929737, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 04:28:57.608560 18553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:57.597137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45621 (local address 127.0.98.62:34197)
0620 04:28:57.597285 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 04:28:57.597289 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:28:57.598103 (+ 814us) server_negotiation.cc:407] Connection header received
0620 04:28:57.599615 (+ 1512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:28:57.599618 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:28:57.599674 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:28:57.599746 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:28:57.601066 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:28:57.601602 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:28:57.602753 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:28:57.602978 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:28:57.603065 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:28:57.605315 (+ 2250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:28:57.605319 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:28:57.605321 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:28:57.605347 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:28:57.606668 (+ 1321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:28:57.607136 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:28:57.607139 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:28:57.607141 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:28:57.607215 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:28:57.607558 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:28:57.607561 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:28:57.607562 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:28:57.607739 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:28:57.607828 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 04:28:57.608340 (+ 512us) server_negotiation.cc:299] Negotiation successful
0620 04:28:57.608443 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 04:28:57.609378 18841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:57.597523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34197 (local address 127.0.98.2:45621)
0620 04:28:57.597962 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0620 04:28:57.598004 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 04:28:57.599414 (+ 1410us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:28:57.599924 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:28:57.599931 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:28:57.600304 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:28:57.600933 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:28:57.600952 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:28:57.601711 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:28:57.601714 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:28:57.602641 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:28:57.602649 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:28:57.603108 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:28:57.603112 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:28:57.603262 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:28:57.603772 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:28:57.603795 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:28:57.605158 (+ 1363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:28:57.606784 (+ 1626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:28:57.606790 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:28:57.606804 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:28:57.607044 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:28:57.607309 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:28:57.607312 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:28:57.607314 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:28:57.607470 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:28:57.607826 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:28:57.607832 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:28:57.608112 (+ 280us) client_negotiation.cc:769] Sending connection context
0620 04:28:57.608284 (+ 172us) client_negotiation.cc:240] Negotiation successful
0620 04:28:57.608558 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":119,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260620 04:28:57.610487 18839 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34197
I20260620 04:28:57.610714 18839 heartbeater.cc:464] Registering TS with master...
I20260620 04:28:57.611277 18839 heartbeater.cc:511] Master 127.0.98.62:34197 requested a full tablet report, sending...
I20260620 04:28:57.612475 18509 ts_manager.cc:194] Registered new tserver with Master: daca0dbd55b64835865f0a771657e390 (127.0.98.2:40669)
I20260620 04:28:57.613189 18509 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:45621
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929496330398-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929496330398-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 04:28:57.622812 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929496330398-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929496330398-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929496330398-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929496330398-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34197
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929496330398-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:28:57.727869 18844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:28:57.728134 18844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:28:57.728174 18844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:28:57.732173 18844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:28:57.732229 18844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:28:57.732286 18844 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 04:28:57.737536 18844 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929496330398-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929496330398-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929496330398-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929496330398-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34197
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18844
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929496330398-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:28:57.738492 18844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:28:57.739420 18844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:28:57.748194 18850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:28:57.748219 18849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:28:57.748219 18852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:28:57.748543 18844 server_base.cc:1034] running on GCE node
I20260620 04:28:57.748905 18844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:28:57.749514 18844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:28:57.750667 18844 hybrid_clock.cc:648] HybridClock initialized: now 1781929737750648 us; error 30 us; skew 500 ppm
Jun 20 04:28:57 dist-test-slave-nk6d krb5kdc[18469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929737, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:28:57.753265 18844 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 04:28:57.754051 18844 webserver.cc:460] Webserver started at http://127.0.98.3:45223/ using document root <none> and password file <none>
I20260620 04:28:57.754552 18844 fs_manager.cc:359] Metadata directory not provided
I20260620 04:28:57.754611 18844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:28:57.754788 18844 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:28:57.756745 18844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929496330398-392-0/minicluster-data/ts-2/data/instance:
uuid: "05f390d10a974005a72f773ebce33e08"
format_stamp: "Formatted at 2026-06-20 04:28:57 on dist-test-slave-nk6d"
server_key: "a3c71c394570d3ed7811e0ee3c0c8bc6"
server_key_iv: "ff1b558595b3c11c015543e69ea87e95"
server_key_version: "encryptionkey@0"
I20260620 04:28:57.757236 18844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929496330398-392-0/minicluster-data/ts-2/wal/instance:
uuid: "05f390d10a974005a72f773ebce33e08"
format_stamp: "Formatted at 2026-06-20 04:28:57 on dist-test-slave-nk6d"
server_key: "a3c71c394570d3ed7811e0ee3c0c8bc6"
server_key_iv: "ff1b558595b3c11c015543e69ea87e95"
server_key_version: "encryptionkey@0"
I20260620 04:28:57.760588 18844 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 04:28:57.762892 18859 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:28:57.764142 18844 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:28:57.764243 18844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929496330398-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929496330398-392-0/minicluster-data/ts-2/wal
uuid: "05f390d10a974005a72f773ebce33e08"
format_stamp: "Formatted at 2026-06-20 04:28:57 on dist-test-slave-nk6d"
server_key: "a3c71c394570d3ed7811e0ee3c0c8bc6"
server_key_iv: "ff1b558595b3c11c015543e69ea87e95"
server_key_version: "encryptionkey@0"
I20260620 04:28:57.764340 18844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929496330398-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929496330398-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929496330398-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:28:57.796731 18844 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:28:57.799868 18844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:28:57.800072 18844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:28:57.800661 18844 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:28:57.801450 18844 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:28:57.801494 18844 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:28:57.801545 18844 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:28:57.801571 18844 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:28:57.809692 18844 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45353
I20260620 04:28:57.809726 18972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45353 every 8 connection(s)
I20260620 04:28:57.810549 18844 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929496330398-392-0/minicluster-data/ts-2/data/info.pb
I20260620 04:28:57.818753 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 18844
I20260620 04:28:57.818882 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929496330398-392-0/minicluster-data/ts-2/wal/instance
I20260620 04:28:57.819170 392 external_mini_cluster.cc:1428] Setting key 89ed36136f5af9c7523bcac41626a1ec
Jun 20 04:28:57 dist-test-slave-nk6d krb5kdc[18469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929737, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:28:57.823499 18553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:57.812280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56205 (local address 127.0.98.62:34197)
0620 04:28:57.812437 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 04:28:57.812440 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:28:57.813264 (+ 824us) server_negotiation.cc:407] Connection header received
0620 04:28:57.814572 (+ 1308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:28:57.814575 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:28:57.814631 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:28:57.814703 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:28:57.815845 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:28:57.816428 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:28:57.817542 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:28:57.817758 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:28:57.817842 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:28:57.820516 (+ 2674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:28:57.820520 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:28:57.820523 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:28:57.820552 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:28:57.821896 (+ 1344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:28:57.822316 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:28:57.822318 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:28:57.822320 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:28:57.822402 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:28:57.822643 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:28:57.822645 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:28:57.822646 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:28:57.822813 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:28:57.822882 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 04:28:57.823272 (+ 390us) server_negotiation.cc:299] Negotiation successful
0620 04:28:57.823385 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 04:28:57.824319 18975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:57.812681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34197 (local address 127.0.98.3:56205)
0620 04:28:57.813121 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0620 04:28:57.813162 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 04:28:57.814364 (+ 1202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:28:57.814890 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:28:57.814896 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:28:57.815236 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:28:57.815699 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:28:57.815711 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:28:57.816540 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:28:57.816543 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:28:57.817429 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:28:57.817438 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:28:57.817895 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:28:57.817899 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:28:57.818054 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:28:57.818596 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:28:57.818616 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:28:57.820336 (+ 1720us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:28:57.821999 (+ 1663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:28:57.822004 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:28:57.822015 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:28:57.822245 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:28:57.822464 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:28:57.822466 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:28:57.822467 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:28:57.822583 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:28:57.822882 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:28:57.822886 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:28:57.823100 (+ 214us) client_negotiation.cc:769] Sending connection context
0620 04:28:57.823243 (+ 143us) client_negotiation.cc:240] Negotiation successful
0620 04:28:57.823501 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":115,"threads_started":1}
I20260620 04:28:57.825340 18973 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34197
I20260620 04:28:57.825568 18973 heartbeater.cc:464] Registering TS with master...
I20260620 04:28:57.826053 18973 heartbeater.cc:511] Master 127.0.98.62:34197 requested a full tablet report, sending...
I20260620 04:28:57.827042 18509 ts_manager.cc:194] Registered new tserver with Master: 05f390d10a974005a72f773ebce33e08 (127.0.98.3:45353)
I20260620 04:28:57.827627 18509 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:56205
I20260620 04:28:57.832891 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 04:28:57.841217 18553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:57.834616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53190 (local address 127.0.98.62:34197)
0620 04:28:57.834754 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 04:28:57.834758 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:28:57.834911 (+ 153us) server_negotiation.cc:407] Connection header received
0620 04:28:57.835027 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:28:57.835029 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:28:57.835068 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:28:57.835123 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:28:57.835701 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:28:57.836282 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:28:57.837032 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:28:57.837239 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:28:57.837308 (+ 69us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:28:57.838627 (+ 1319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:28:57.838630 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:28:57.838632 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:28:57.838654 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:28:57.839889 (+ 1235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:28:57.840294 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:28:57.840296 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:28:57.840297 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:28:57.840366 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:28:57.840567 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:28:57.840568 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:28:57.840569 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:28:57.840742 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:28:57.840874 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 04:28:57.840996 (+ 122us) server_negotiation.cc:299] Negotiation successful
0620 04:28:57.841064 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 04:28:57.843848 18509 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53190:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 04:28:57.846707 18509 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 04:28:57.859946 18985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:57.854566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37762 (local address 127.0.98.1:36563)
0620 04:28:57.855179 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0620 04:28:57.855190 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 04:28:57.855212 (+ 22us) server_negotiation.cc:407] Connection header received
0620 04:28:57.855286 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:28:57.855293 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:28:57.855499 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:28:57.855684 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:28:57.856054 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:28:57.857024 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:28:57.858432 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:28:57.859240 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:28:57.859382 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:28:57.859532 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 04:28:57.859703 (+ 171us) server_negotiation.cc:299] Negotiation successful
0620 04:28:57.859797 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513,"thread_start_us":152,"threads_started":1}
I20260620 04:28:57.859918 18984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:57.854393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36563 (local address 127.0.0.1:37762)
0620 04:28:57.854712 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0620 04:28:57.854761 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 04:28:57.854889 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:28:57.855698 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:28:57.855705 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:28:57.855727 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:28:57.855873 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:28:57.855883 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:28:57.857173 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:28:57.857176 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:28:57.858305 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:28:57.858317 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:28:57.859404 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:28:57.859408 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:28:57.859526 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:28:57.859566 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 04:28:57.859663 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 04:28:57.859778 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":81,"threads_started":1}
I20260620 04:28:57.860723 18987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:57.855161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45353 (local address 127.0.0.1:46224)
0620 04:28:57.855450 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0620 04:28:57.855463 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:28:57.855542 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:28:57.856454 (+ 912us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:28:57.856457 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:28:57.856478 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:28:57.856573 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:28:57.856580 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:28:57.857949 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:28:57.857952 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:28:57.859065 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:28:57.859076 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:28:57.860403 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:28:57.860407 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:28:57.860505 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:28:57.860529 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 04:28:57.860578 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 04:28:57.860627 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"mutex_wait_us":42,"thread_start_us":85,"threads_started":1}
I20260620 04:28:57.860888 18989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:57.855635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46224 (local address 127.0.98.3:45353)
0620 04:28:57.856039 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0620 04:28:57.856050 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 04:28:57.856067 (+ 17us) server_negotiation.cc:407] Connection header received
0620 04:28:57.856129 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:28:57.856134 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:28:57.856312 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:28:57.856459 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:28:57.856684 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:28:57.857822 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:28:57.859177 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:28:57.860265 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:28:57.860386 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:28:57.860553 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0620 04:28:57.860605 (+ 52us) server_negotiation.cc:299] Negotiation successful
0620 04:28:57.860690 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":96,"threads_started":1}
I20260620 04:28:57.861779 18986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:57.854710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40669 (local address 127.0.0.1:60324)
0620 04:28:57.854987 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0620 04:28:57.855002 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 04:28:57.855098 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:28:57.856620 (+ 1522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:28:57.856622 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:28:57.856638 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:28:57.856735 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:28:57.856741 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:28:57.858041 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:28:57.858044 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:28:57.859176 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:28:57.859187 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:28:57.861353 (+ 2166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:28:57.861357 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:28:57.861451 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:28:57.861476 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 04:28:57.861564 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 04:28:57.861636 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"mutex_wait_us":24,"thread_start_us":96,"threads_started":1}
I20260620 04:28:57.863078 18639 tablet_service.cc:1467] Processing CreateTablet for tablet cb4790504806428d8dc356e760d141ec (DEFAULT_TABLE table=test-table [id=d535488b3bf0406bba4aa8025dce46cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:28:57.863564 18907 tablet_service.cc:1467] Processing CreateTablet for tablet cb4790504806428d8dc356e760d141ec (DEFAULT_TABLE table=test-table [id=d535488b3bf0406bba4aa8025dce46cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:28:57.864250 18639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb4790504806428d8dc356e760d141ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:28:57.864482 18988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:57.855206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60324 (local address 127.0.98.2:40669)
0620 04:28:57.856024 (+ 818us) server_negotiation.cc:206] Beginning negotiation
0620 04:28:57.856035 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 04:28:57.856058 (+ 23us) server_negotiation.cc:407] Connection header received
0620 04:28:57.856129 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:28:57.856134 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:28:57.856312 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:28:57.856459 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:28:57.856884 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:28:57.857935 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:28:57.859312 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:28:57.861028 (+ 1716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:28:57.861135 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:28:57.861287 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 04:28:57.864223 (+ 2936us) server_negotiation.cc:299] Negotiation successful
0620 04:28:57.864362 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":703,"thread_start_us":85,"threads_started":1}
I20260620 04:28:57.864509 18907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb4790504806428d8dc356e760d141ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:28:57.870064 18773 tablet_service.cc:1467] Processing CreateTablet for tablet cb4790504806428d8dc356e760d141ec (DEFAULT_TABLE table=test-table [id=d535488b3bf0406bba4aa8025dce46cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:28:57.871074 18773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb4790504806428d8dc356e760d141ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:28:57.871302 18990 tablet_bootstrap.cc:492] T cb4790504806428d8dc356e760d141ec P e62476cda7b94339abcd5c3df544cfdd: Bootstrap starting.
I20260620 04:28:57.872305 18991 tablet_bootstrap.cc:492] T cb4790504806428d8dc356e760d141ec P 05f390d10a974005a72f773ebce33e08: Bootstrap starting.
I20260620 04:28:57.873528 18990 tablet_bootstrap.cc:654] T cb4790504806428d8dc356e760d141ec P e62476cda7b94339abcd5c3df544cfdd: Neither blocks nor log segments found. Creating new log.
I20260620 04:28:57.873878 18991 tablet_bootstrap.cc:654] T cb4790504806428d8dc356e760d141ec P 05f390d10a974005a72f773ebce33e08: Neither blocks nor log segments found. Creating new log.
I20260620 04:28:57.874749 18990 log.cc:824] T cb4790504806428d8dc356e760d141ec P e62476cda7b94339abcd5c3df544cfdd: Log is configured to *not* fsync() on all Append() calls
I20260620 04:28:57.874784 18991 log.cc:824] T cb4790504806428d8dc356e760d141ec P 05f390d10a974005a72f773ebce33e08: Log is configured to *not* fsync() on all Append() calls
I20260620 04:28:57.876458 18991 tablet_bootstrap.cc:492] T cb4790504806428d8dc356e760d141ec P 05f390d10a974005a72f773ebce33e08: No bootstrap required, opened a new log
I20260620 04:28:57.876636 18991 ts_tablet_manager.cc:1397] T cb4790504806428d8dc356e760d141ec P 05f390d10a974005a72f773ebce33e08: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260620 04:28:57.877920 18994 tablet_bootstrap.cc:492] T cb4790504806428d8dc356e760d141ec P daca0dbd55b64835865f0a771657e390: Bootstrap starting.
I20260620 04:28:57.879616 18994 tablet_bootstrap.cc:654] T cb4790504806428d8dc356e760d141ec P daca0dbd55b64835865f0a771657e390: Neither blocks nor log segments found. Creating new log.
I20260620 04:28:57.879750 18990 tablet_bootstrap.cc:492] T cb4790504806428d8dc356e760d141ec P e62476cda7b94339abcd5c3df544cfdd: No bootstrap required, opened a new log
I20260620 04:28:57.879722 18991 raft_consensus.cc:348] T cb4790504806428d8dc356e760d141ec P 05f390d10a974005a72f773ebce33e08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e62476cda7b94339abcd5c3df544cfdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36563 } } peers { permanent_uuid: "05f390d10a974005a72f773ebce33e08" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45353 } } peers { permanent_uuid: "daca0dbd55b64835865f0a771657e390" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40669 } }
I20260620 04:28:57.879933 18990 ts_tablet_manager.cc:1397] T cb4790504806428d8dc356e760d141ec P e62476cda7b94339abcd5c3df544cfdd: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20260620 04:28:57.880015 18991 raft_consensus.cc:374] T cb4790504806428d8dc356e760d141ec P 05f390d10a974005a72f773ebce33e08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:28:57.880072 18991 raft_consensus.cc:729] T cb4790504806428d8dc356e760d141ec P 05f390d10a974005a72f773ebce33e08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05f390d10a974005a72f773ebce33e08, State: Initialized, Role: FOLLOWER
I20260620 04:28:57.880524 18991 consensus_queue.cc:260] T cb4790504806428d8dc356e760d141ec P 05f390d10a974005a72f773ebce33e08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e62476cda7b94339abcd5c3df544cfdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36563 } } peers { permanent_uuid: "05f390d10a974005a72f773ebce33e08" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45353 } } peers { permanent_uuid: "daca0dbd55b64835865f0a771657e390" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40669 } }
I20260620 04:28:57.880748 18994 log.cc:824] T cb4790504806428d8dc356e760d141ec P daca0dbd55b64835865f0a771657e390: Log is configured to *not* fsync() on all Append() calls
I20260620 04:28:57.881199 18973 heartbeater.cc:503] Master 127.0.98.62:34197 was elected leader, sending a full tablet report...
I20260620 04:28:57.881461 18991 ts_tablet_manager.cc:1428] T cb4790504806428d8dc356e760d141ec P 05f390d10a974005a72f773ebce33e08: Time spent starting tablet: real 0.005s user 0.001s sys 0.004s
I20260620 04:28:57.883586 18994 tablet_bootstrap.cc:492] T cb4790504806428d8dc356e760d141ec P daca0dbd55b64835865f0a771657e390: No bootstrap required, opened a new log
I20260620 04:28:57.883764 18994 ts_tablet_manager.cc:1397] T cb4790504806428d8dc356e760d141ec P daca0dbd55b64835865f0a771657e390: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20260620 04:28:57.883675 18990 raft_consensus.cc:348] T cb4790504806428d8dc356e760d141ec P e62476cda7b94339abcd5c3df544cfdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e62476cda7b94339abcd5c3df544cfdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36563 } } peers { permanent_uuid: "05f390d10a974005a72f773ebce33e08" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45353 } } peers { permanent_uuid: "daca0dbd55b64835865f0a771657e390" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40669 } }
I20260620 04:28:57.883926 18990 raft_consensus.cc:374] T cb4790504806428d8dc356e760d141ec P e62476cda7b94339abcd5c3df544cfdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:28:57.884008 18990 raft_consensus.cc:729] T cb4790504806428d8dc356e760d141ec P e62476cda7b94339abcd5c3df544cfdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e62476cda7b94339abcd5c3df544cfdd, State: Initialized, Role: FOLLOWER
I20260620 04:28:57.884466 18990 consensus_queue.cc:260] T cb4790504806428d8dc356e760d141ec P e62476cda7b94339abcd5c3df544cfdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e62476cda7b94339abcd5c3df544cfdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36563 } } peers { permanent_uuid: "05f390d10a974005a72f773ebce33e08" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45353 } } peers { permanent_uuid: "daca0dbd55b64835865f0a771657e390" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40669 } }
I20260620 04:28:57.885047 18705 heartbeater.cc:503] Master 127.0.98.62:34197 was elected leader, sending a full tablet report...
I20260620 04:28:57.885442 18990 ts_tablet_manager.cc:1428] T cb4790504806428d8dc356e760d141ec P e62476cda7b94339abcd5c3df544cfdd: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260620 04:28:57.886430 18706 tablet.cc:2369] T cb4790504806428d8dc356e760d141ec P e62476cda7b94339abcd5c3df544cfdd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 04:28:57.887807 18994 raft_consensus.cc:348] T cb4790504806428d8dc356e760d141ec P daca0dbd55b64835865f0a771657e390 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e62476cda7b94339abcd5c3df544cfdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36563 } } peers { permanent_uuid: "05f390d10a974005a72f773ebce33e08" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45353 } } peers { permanent_uuid: "daca0dbd55b64835865f0a771657e390" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40669 } }
I20260620 04:28:57.888067 18994 raft_consensus.cc:374] T cb4790504806428d8dc356e760d141ec P daca0dbd55b64835865f0a771657e390 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:28:57.888123 18994 raft_consensus.cc:729] T cb4790504806428d8dc356e760d141ec P daca0dbd55b64835865f0a771657e390 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: daca0dbd55b64835865f0a771657e390, State: Initialized, Role: FOLLOWER
I20260620 04:28:57.888532 18994 consensus_queue.cc:260] T cb4790504806428d8dc356e760d141ec P daca0dbd55b64835865f0a771657e390 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e62476cda7b94339abcd5c3df544cfdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36563 } } peers { permanent_uuid: "05f390d10a974005a72f773ebce33e08" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45353 } } peers { permanent_uuid: "daca0dbd55b64835865f0a771657e390" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40669 } }
I20260620 04:28:57.889129 18839 heartbeater.cc:503] Master 127.0.98.62:34197 was elected leader, sending a full tablet report...
I20260620 04:28:57.889372 18994 ts_tablet_manager.cc:1428] T cb4790504806428d8dc356e760d141ec P daca0dbd55b64835865f0a771657e390: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260620 04:28:57.964008 18998 raft_consensus.cc:482] T cb4790504806428d8dc356e760d141ec P daca0dbd55b64835865f0a771657e390 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:28:57.964195 18998 raft_consensus.cc:504] T cb4790504806428d8dc356e760d141ec P daca0dbd55b64835865f0a771657e390 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e62476cda7b94339abcd5c3df544cfdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36563 } } peers { permanent_uuid: "05f390d10a974005a72f773ebce33e08" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45353 } } peers { permanent_uuid: "daca0dbd55b64835865f0a771657e390" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40669 } }
I20260620 04:28:57.965248 18998 leader_election.cc:288] T cb4790504806428d8dc356e760d141ec P daca0dbd55b64835865f0a771657e390 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e62476cda7b94339abcd5c3df544cfdd (127.0.98.1:36563), 05f390d10a974005a72f773ebce33e08 (127.0.98.3:45353)
I20260620 04:28:57.968845 18841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:57.965511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36563 (local address 127.0.98.2:35367)
0620 04:28:57.965636 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 04:28:57.965652 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 04:28:57.965743 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:28:57.966121 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:28:57.966124 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:28:57.966141 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:28:57.966233 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:28:57.966238 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:28:57.966937 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:28:57.966940 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:28:57.967790 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:28:57.967799 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:28:57.968458 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:28:57.968461 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:28:57.968563 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:28:57.968581 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 04:28:57.968643 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 04:28:57.968712 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":17}
I20260620 04:28:57.968845 18985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:57.965572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35367 (local address 127.0.98.1:36563)
0620 04:28:57.965706 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 04:28:57.965711 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:28:57.965727 (+ 16us) server_negotiation.cc:407] Connection header received
0620 04:28:57.965850 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:28:57.965854 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:28:57.965903 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:28:57.965994 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:28:57.966341 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:28:57.966829 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:28:57.967905 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:28:57.968352 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:28:57.968443 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:28:57.968525 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 04:28:57.968671 (+ 146us) server_negotiation.cc:299] Negotiation successful
0620 04:28:57.968737 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 04:28:57.969306 18989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:57.965963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35075 (local address 127.0.98.3:45353)
0620 04:28:57.966089 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 04:28:57.966094 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:28:57.966178 (+ 84us) server_negotiation.cc:407] Connection header received
0620 04:28:57.966305 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:28:57.966308 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:28:57.966351 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:28:57.966408 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:28:57.966640 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:28:57.967169 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:28:57.968318 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:28:57.968818 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:28:57.968925 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:28:57.968982 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 04:28:57.969155 (+ 173us) server_negotiation.cc:299] Negotiation successful
0620 04:28:57.969216 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 04:28:57.969321 18999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:57.965785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45353 (local address 127.0.98.2:35075)
0620 04:28:57.966100 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0620 04:28:57.966112 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 04:28:57.966205 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:28:57.966424 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:28:57.966427 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:28:57.966441 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:28:57.966549 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:28:57.966554 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:28:57.967261 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:28:57.967264 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:28:57.968167 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:28:57.968176 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:28:57.968946 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:28:57.968950 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:28:57.969037 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:28:57.969058 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 04:28:57.969123 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 04:28:57.969181 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"mutex_wait_us":20,"thread_start_us":125,"threads_started":1}
I20260620 04:28:57.969786 18659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4790504806428d8dc356e760d141ec" candidate_uuid: "daca0dbd55b64835865f0a771657e390" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e62476cda7b94339abcd5c3df544cfdd" is_pre_election: true
I20260620 04:28:57.970115 18659 raft_consensus.cc:2454] T cb4790504806428d8dc356e760d141ec P e62476cda7b94339abcd5c3df544cfdd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate daca0dbd55b64835865f0a771657e390 in term 0.
I20260620 04:28:57.970134 18927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4790504806428d8dc356e760d141ec" candidate_uuid: "daca0dbd55b64835865f0a771657e390" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f390d10a974005a72f773ebce33e08" is_pre_election: true
I20260620 04:28:57.970412 18927 raft_consensus.cc:2454] T cb4790504806428d8dc356e760d141ec P 05f390d10a974005a72f773ebce33e08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate daca0dbd55b64835865f0a771657e390 in term 0.
I20260620 04:28:57.970614 18726 leader_election.cc:302] T cb4790504806428d8dc356e760d141ec P daca0dbd55b64835865f0a771657e390 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: daca0dbd55b64835865f0a771657e390, e62476cda7b94339abcd5c3df544cfdd; no voters:
I20260620 04:28:57.970875 18998 raft_consensus.cc:2781] T cb4790504806428d8dc356e760d141ec P daca0dbd55b64835865f0a771657e390 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 04:28:57.970971 18998 raft_consensus.cc:482] T cb4790504806428d8dc356e760d141ec P daca0dbd55b64835865f0a771657e390 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 04:28:57.971004 18998 raft_consensus.cc:3037] T cb4790504806428d8dc356e760d141ec P daca0dbd55b64835865f0a771657e390 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:28:57.972026 18998 raft_consensus.cc:504] T cb4790504806428d8dc356e760d141ec P daca0dbd55b64835865f0a771657e390 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e62476cda7b94339abcd5c3df544cfdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36563 } } peers { permanent_uuid: "05f390d10a974005a72f773ebce33e08" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45353 } } peers { permanent_uuid: "daca0dbd55b64835865f0a771657e390" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40669 } }
I20260620 04:28:57.972328 18998 leader_election.cc:288] T cb4790504806428d8dc356e760d141ec P daca0dbd55b64835865f0a771657e390 [CANDIDATE]: Term 1 election: Requested vote from peers e62476cda7b94339abcd5c3df544cfdd (127.0.98.1:36563), 05f390d10a974005a72f773ebce33e08 (127.0.98.3:45353)
I20260620 04:28:57.972712 18659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4790504806428d8dc356e760d141ec" candidate_uuid: "daca0dbd55b64835865f0a771657e390" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e62476cda7b94339abcd5c3df544cfdd"
I20260620 04:28:57.972740 18927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4790504806428d8dc356e760d141ec" candidate_uuid: "daca0dbd55b64835865f0a771657e390" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f390d10a974005a72f773ebce33e08"
I20260620 04:28:57.972846 18659 raft_consensus.cc:3037] T cb4790504806428d8dc356e760d141ec P e62476cda7b94339abcd5c3df544cfdd [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:28:57.972893 18927 raft_consensus.cc:3037] T cb4790504806428d8dc356e760d141ec P 05f390d10a974005a72f773ebce33e08 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:28:57.974063 18659 raft_consensus.cc:2454] T cb4790504806428d8dc356e760d141ec P e62476cda7b94339abcd5c3df544cfdd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate daca0dbd55b64835865f0a771657e390 in term 1.
I20260620 04:28:57.974063 18927 raft_consensus.cc:2454] T cb4790504806428d8dc356e760d141ec P 05f390d10a974005a72f773ebce33e08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate daca0dbd55b64835865f0a771657e390 in term 1.
I20260620 04:28:57.974337 18726 leader_election.cc:302] T cb4790504806428d8dc356e760d141ec P daca0dbd55b64835865f0a771657e390 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: daca0dbd55b64835865f0a771657e390, e62476cda7b94339abcd5c3df544cfdd; no voters:
I20260620 04:28:57.974484 18998 raft_consensus.cc:2781] T cb4790504806428d8dc356e760d141ec P daca0dbd55b64835865f0a771657e390 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:28:57.974687 18998 raft_consensus.cc:686] T cb4790504806428d8dc356e760d141ec P daca0dbd55b64835865f0a771657e390 [term 1 LEADER]: Becoming Leader. State: Replica: daca0dbd55b64835865f0a771657e390, State: Running, Role: LEADER
I20260620 04:28:57.975037 18998 consensus_queue.cc:237] T cb4790504806428d8dc356e760d141ec P daca0dbd55b64835865f0a771657e390 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e62476cda7b94339abcd5c3df544cfdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36563 } } peers { permanent_uuid: "05f390d10a974005a72f773ebce33e08" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45353 } } peers { permanent_uuid: "daca0dbd55b64835865f0a771657e390" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40669 } }
I20260620 04:28:57.978034 18509 catalog_manager.cc:5495] T cb4790504806428d8dc356e760d141ec P daca0dbd55b64835865f0a771657e390 reported cstate change: term changed from 0 to 1, leader changed from <none> to daca0dbd55b64835865f0a771657e390 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "daca0dbd55b64835865f0a771657e390" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e62476cda7b94339abcd5c3df544cfdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05f390d10a974005a72f773ebce33e08" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45353 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "daca0dbd55b64835865f0a771657e390" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40669 } health_report { overall_health: HEALTHY } } }
I20260620 04:28:58.011476 18988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:58.007886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60330 (local address 127.0.98.2:40669)
0620 04:28:58.008063 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 04:28:58.008066 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:28:58.008080 (+ 14us) server_negotiation.cc:407] Connection header received
0620 04:28:58.008203 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:28:58.008205 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:28:58.008255 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:28:58.008332 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:28:58.008617 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:28:58.009132 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:28:58.010000 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:28:58.010222 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:28:58.010312 (+ 90us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:28:58.010613 (+ 301us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:28:58.011057 (+ 444us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:28:58.011169 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 04:28:58.011295 (+ 126us) server_negotiation.cc:299] Negotiation successful
0620 04:28:58.011330 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260620 04:28:58.017534 18659 raft_consensus.cc:1261] T cb4790504806428d8dc356e760d141ec P e62476cda7b94339abcd5c3df544cfdd [term 1 FOLLOWER]: Refusing update from remote peer daca0dbd55b64835865f0a771657e390: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:28:58.017534 18927 raft_consensus.cc:1261] T cb4790504806428d8dc356e760d141ec P 05f390d10a974005a72f773ebce33e08 [term 1 FOLLOWER]: Refusing update from remote peer daca0dbd55b64835865f0a771657e390: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:28:58.018297 18998 consensus_queue.cc:1035] T cb4790504806428d8dc356e760d141ec P daca0dbd55b64835865f0a771657e390 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e62476cda7b94339abcd5c3df544cfdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:28:58.018501 19000 consensus_queue.cc:1035] T cb4790504806428d8dc356e760d141ec P daca0dbd55b64835865f0a771657e390 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f390d10a974005a72f773ebce33e08" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:28:58.027194 19004 mvcc.cc:205] Tried to move back new op lower bound from 7298784206912102400 to 7298784206747930624. Current Snapshot: MvccSnapshot[applied={T|T < 7298784206912102400}]
I20260620 04:28:58.036434 18509 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53190:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:28:58.036592 18509 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53190:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:28:58.038878 18509 catalog_manager.cc:5782] T 00000000000000000000000000000000 P de4c89a3d70a491ba1bce513a4a7f6de: Sending DeleteTablet for 3 replicas of tablet cb4790504806428d8dc356e760d141ec
I20260620 04:28:58.039728 18907 tablet_service.cc:1514] Processing DeleteTablet for tablet cb4790504806428d8dc356e760d141ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:28:58 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46224
I20260620 04:28:58.039712 18639 tablet_service.cc:1514] Processing DeleteTablet for tablet cb4790504806428d8dc356e760d141ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:28:58 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37762
I20260620 04:28:58.040434 19014 tablet_replica.cc:330] T cb4790504806428d8dc356e760d141ec P 05f390d10a974005a72f773ebce33e08: stopping tablet replica
I20260620 04:28:58.040427 18773 tablet_service.cc:1514] Processing DeleteTablet for tablet cb4790504806428d8dc356e760d141ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:28:58 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60324
I20260620 04:28:58.040539 19013 tablet_replica.cc:330] T cb4790504806428d8dc356e760d141ec P e62476cda7b94339abcd5c3df544cfdd: stopping tablet replica
I20260620 04:28:58.040863 19013 raft_consensus.cc:2229] T cb4790504806428d8dc356e760d141ec P e62476cda7b94339abcd5c3df544cfdd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:28:58.040861 19014 raft_consensus.cc:2229] T cb4790504806428d8dc356e760d141ec P 05f390d10a974005a72f773ebce33e08 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:28:58.040948 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 18576
I20260620 04:28:58.041172 19013 raft_consensus.cc:2258] T cb4790504806428d8dc356e760d141ec P e62476cda7b94339abcd5c3df544cfdd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:28:58.042335 19014 raft_consensus.cc:2258] T cb4790504806428d8dc356e760d141ec P 05f390d10a974005a72f773ebce33e08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:28:58.043646 19014 ts_tablet_manager.cc:1905] T cb4790504806428d8dc356e760d141ec P 05f390d10a974005a72f773ebce33e08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:28:58.045784 19014 ts_tablet_manager.cc:1918] T cb4790504806428d8dc356e760d141ec P 05f390d10a974005a72f773ebce33e08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:28:58.045866 19014 log.cc:1196] T cb4790504806428d8dc356e760d141ec P 05f390d10a974005a72f773ebce33e08: Deleting WAL directory at /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929496330398-392-0/minicluster-data/ts-2/wal/wals/cb4790504806428d8dc356e760d141ec
I20260620 04:28:58.046129 19014 ts_tablet_manager.cc:1939] T cb4790504806428d8dc356e760d141ec P 05f390d10a974005a72f773ebce33e08: Deleting consensus metadata
I20260620 04:28:58.047178 18498 catalog_manager.cc:4841] TS 05f390d10a974005a72f773ebce33e08 (127.0.98.3:45353): tablet cb4790504806428d8dc356e760d141ec (table test-table [id=d535488b3bf0406bba4aa8025dce46cc]) successfully deleted
W20260620 04:28:58.048097 18498 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36563 (error 108)
I20260620 04:28:58.048540 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 18710
I20260620 04:28:58.049960 18986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:58.049720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36563 (local address 127.0.0.1:37774)
0620 04:28:58.049830 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0620 04:28:58.049890 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260620 04:28:58.050123 18498 catalog_manager.cc:4584] TS e62476cda7b94339abcd5c3df544cfdd (127.0.98.1:36563): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cb4790504806428d8dc356e760d141ec: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36563: connect: Connection refused (error 111)
I20260620 04:28:58.053720 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 18844
I20260620 04:28:58.054638 18986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:58.053869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40669 (local address 127.0.0.1:60344)
0620 04:28:58.054505 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0620 04:28:58.054559 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:40669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":560}
I20260620 04:28:58.058295 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 18482
2026-06-20T04:28:58Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2184 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-20T04:28:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:28:58Z Disabled control of system clock
I20260620 04:28:58.085469 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929496330398-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929496330398-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929496330398-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35035
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:46365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35035
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 04:28:58.191337 19020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:28:58.191570 19020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:28:58.191601 19020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:28:58.195526 19020 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:28:58.195583 19020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:28:58.195597 19020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:28:58.195612 19020 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:28:58.195640 19020 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:28:58.200343 19020 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929496330398-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929496330398-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35035
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35035
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19020
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929496330398-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:28:58.201373 19020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:28:58.202716 19020 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:28:58.210909 19026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:28:58.210928 19028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:28:58.210995 19025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:28:58.211251 19020 server_base.cc:1034] running on GCE node
I20260620 04:28:58.212083 19020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:28:58.213147 19020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:28:58.214313 19020 hybrid_clock.cc:648] HybridClock initialized: now 1781929738214289 us; error 38 us; skew 500 ppm
I20260620 04:28:58.216277 19020 webserver.cc:460] Webserver started at http://127.0.98.62:43607/ using document root <none> and password file <none>
I20260620 04:28:58.216812 19020 fs_manager.cc:359] Metadata directory not provided
I20260620 04:28:58.216877 19020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:28:58.217067 19020 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:28:58.219012 19020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929496330398-392-0/minicluster-data/master-0/data/instance:
uuid: "75acd78b79304fe08d8d0d04e816de34"
format_stamp: "Formatted at 2026-06-20 04:28:58 on dist-test-slave-nk6d"
server_key: "f8e12727b97eb56e7f8cf7c057f9e22f"
server_key_iv: "482f8b9c2acd19f197f3fa8be4947115"
server_key_version: "encryptionkey@0"
I20260620 04:28:58.219509 19020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929496330398-392-0/minicluster-data/master-0/wal/instance:
uuid: "75acd78b79304fe08d8d0d04e816de34"
format_stamp: "Formatted at 2026-06-20 04:28:58 on dist-test-slave-nk6d"
server_key: "f8e12727b97eb56e7f8cf7c057f9e22f"
server_key_iv: "482f8b9c2acd19f197f3fa8be4947115"
server_key_version: "encryptionkey@0"
I20260620 04:28:58.223001 19020 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.006s
I20260620 04:28:58.225297 19034 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:28:58.226483 19020 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 04:28:58.226593 19020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929496330398-392-0/minicluster-data/master-0/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929496330398-392-0/minicluster-data/master-0/wal
uuid: "75acd78b79304fe08d8d0d04e816de34"
format_stamp: "Formatted at 2026-06-20 04:28:58 on dist-test-slave-nk6d"
server_key: "f8e12727b97eb56e7f8cf7c057f9e22f"
server_key_iv: "482f8b9c2acd19f197f3fa8be4947115"
server_key_version: "encryptionkey@0"
I20260620 04:28:58.226693 19020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929496330398-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929496330398-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929496330398-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:28:58.241060 19020 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:28:58.241673 19020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:28:58.241844 19020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:28:58.248224 19020 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35035
I20260620 04:28:58.248236 19086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35035 every 8 connection(s)
I20260620 04:28:58.249110 19020 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:28:58.250589 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 19020
I20260620 04:28:58.250713 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929496330398-392-0/minicluster-data/master-0/wal/instance
I20260620 04:28:58.251060 392 external_mini_cluster.cc:1428] Setting key d2cb0d0d93549f4455a6ddea7dd3c805
I20260620 04:28:58.252532 19087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:28:58.257761 19087 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75acd78b79304fe08d8d0d04e816de34: Bootstrap starting.
I20260620 04:28:58.259627 19090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:58.252410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39404 (local address 127.0.98.62:35035)
0620 04:28:58.253066 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0620 04:28:58.253081 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 04:28:58.253115 (+ 34us) server_negotiation.cc:407] Connection header received
0620 04:28:58.253733 (+ 618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:28:58.253749 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:28:58.254131 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:28:58.254449 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:28:58.254866 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:28:58.255791 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:28:58.256697 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:28:58.257152 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:28:58.257292 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:28:58.258059 (+ 767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:28:58.258064 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:28:58.258075 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:28:58.258441 (+ 366us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:28:58.258493 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:28:58.258511 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:28:58.258592 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 04:28:58.258717 (+ 125us) server_negotiation.cc:299] Negotiation successful
0620 04:28:58.258956 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":120,"threads_started":1}
I20260620 04:28:58.259877 19087 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 75acd78b79304fe08d8d0d04e816de34: Neither blocks nor log segments found. Creating new log.
I20260620 04:28:58.260718 19087 log.cc:824] T 00000000000000000000000000000000 P 75acd78b79304fe08d8d0d04e816de34: Log is configured to *not* fsync() on all Append() calls
I20260620 04:28:58.262549 19087 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75acd78b79304fe08d8d0d04e816de34: No bootstrap required, opened a new log
I20260620 04:28:58.265861 19087 raft_consensus.cc:348] T 00000000000000000000000000000000 P 75acd78b79304fe08d8d0d04e816de34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75acd78b79304fe08d8d0d04e816de34" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35035 } }
I20260620 04:28:58.266077 19087 raft_consensus.cc:374] T 00000000000000000000000000000000 P 75acd78b79304fe08d8d0d04e816de34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:28:58.266135 19087 raft_consensus.cc:729] T 00000000000000000000000000000000 P 75acd78b79304fe08d8d0d04e816de34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75acd78b79304fe08d8d0d04e816de34, State: Initialized, Role: FOLLOWER
I20260620 04:28:58.266546 19087 consensus_queue.cc:260] T 00000000000000000000000000000000 P 75acd78b79304fe08d8d0d04e816de34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75acd78b79304fe08d8d0d04e816de34" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35035 } }
I20260620 04:28:58.266718 19087 raft_consensus.cc:388] T 00000000000000000000000000000000 P 75acd78b79304fe08d8d0d04e816de34 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:28:58.266781 19087 raft_consensus.cc:482] T 00000000000000000000000000000000 P 75acd78b79304fe08d8d0d04e816de34 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:28:58.266876 19087 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 75acd78b79304fe08d8d0d04e816de34 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:28:58.267707 19087 raft_consensus.cc:504] T 00000000000000000000000000000000 P 75acd78b79304fe08d8d0d04e816de34 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75acd78b79304fe08d8d0d04e816de34" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35035 } }
I20260620 04:28:58.268040 19087 leader_election.cc:302] T 00000000000000000000000000000000 P 75acd78b79304fe08d8d0d04e816de34 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 75acd78b79304fe08d8d0d04e816de34; no voters:
I20260620 04:28:58.268307 19087 leader_election.cc:288] T 00000000000000000000000000000000 P 75acd78b79304fe08d8d0d04e816de34 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:28:58.268404 19092 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 75acd78b79304fe08d8d0d04e816de34 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:28:58.268617 19092 raft_consensus.cc:686] T 00000000000000000000000000000000 P 75acd78b79304fe08d8d0d04e816de34 [term 1 LEADER]: Becoming Leader. State: Replica: 75acd78b79304fe08d8d0d04e816de34, State: Running, Role: LEADER
I20260620 04:28:58.268898 19092 consensus_queue.cc:237] T 00000000000000000000000000000000 P 75acd78b79304fe08d8d0d04e816de34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75acd78b79304fe08d8d0d04e816de34" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35035 } }
I20260620 04:28:58.269410 19087 sys_catalog.cc:564] T 00000000000000000000000000000000 P 75acd78b79304fe08d8d0d04e816de34 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:28:58.270426 19094 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75acd78b79304fe08d8d0d04e816de34 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 75acd78b79304fe08d8d0d04e816de34. Latest consensus state: current_term: 1 leader_uuid: "75acd78b79304fe08d8d0d04e816de34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75acd78b79304fe08d8d0d04e816de34" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35035 } } }
I20260620 04:28:58.270546 19094 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75acd78b79304fe08d8d0d04e816de34 [sys.catalog]: This master's current role is: LEADER
I20260620 04:28:58.270408 19093 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75acd78b79304fe08d8d0d04e816de34 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "75acd78b79304fe08d8d0d04e816de34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75acd78b79304fe08d8d0d04e816de34" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35035 } } }
I20260620 04:28:58.270794 19093 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75acd78b79304fe08d8d0d04e816de34 [sys.catalog]: This master's current role is: LEADER
I20260620 04:28:58.270964 19101 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:28:58.274567 19101 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:28:58.280128 19101 catalog_manager.cc:1347] Generated new cluster ID: 6a067a799eb34bdfb49afdfd7537fcce
I20260620 04:28:58.280184 19101 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:28:58.293491 19101 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:28:58.294418 19101 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:28:58.305123 19101 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 75acd78b79304fe08d8d0d04e816de34: Generated new TSK 0
I20260620 04:28:58.305953 19101 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 04:28:58.317234 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929496330398-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929496330398-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929496330398-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929496330398-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35035
--builtin_ntp_servers=127.0.98.20:46365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 04:28:58.424239 19111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:28:58.424490 19111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:28:58.424540 19111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:28:58.428429 19111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:28:58.428479 19111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:28:58.428552 19111 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 04:28:58.433282 19111 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929496330398-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929496330398-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929496330398-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35035
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19111
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929496330398-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:28:58.434320 19111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:28:58.435698 19111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:28:58.445031 19116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:28:58.445094 19119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:28:58.445129 19117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:28:58.445369 19111 server_base.cc:1034] running on GCE node
I20260620 04:28:58.445748 19111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:28:58.446386 19111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:28:58.447543 19111 hybrid_clock.cc:648] HybridClock initialized: now 1781929738447523 us; error 30 us; skew 500 ppm
I20260620 04:28:58.449349 19111 webserver.cc:460] Webserver started at http://127.0.98.1:36291/ using document root <none> and password file <none>
I20260620 04:28:58.449872 19111 fs_manager.cc:359] Metadata directory not provided
I20260620 04:28:58.449939 19111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:28:58.450098 19111 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:28:58.452065 19111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929496330398-392-0/minicluster-data/ts-0/data/instance:
uuid: "3842c22109bd4100b90386d5469e8310"
format_stamp: "Formatted at 2026-06-20 04:28:58 on dist-test-slave-nk6d"
server_key: "cfbe2e829ac8a8944c0f2f7542738232"
server_key_iv: "ad88fb9b761de330a4fa11da3bec3466"
server_key_version: "encryptionkey@0"
I20260620 04:28:58.452600 19111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929496330398-392-0/minicluster-data/ts-0/wal/instance:
uuid: "3842c22109bd4100b90386d5469e8310"
format_stamp: "Formatted at 2026-06-20 04:28:58 on dist-test-slave-nk6d"
server_key: "cfbe2e829ac8a8944c0f2f7542738232"
server_key_iv: "ad88fb9b761de330a4fa11da3bec3466"
server_key_version: "encryptionkey@0"
I20260620 04:28:58.456153 19111 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:28:58.458487 19125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:28:58.459733 19111 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:28:58.459825 19111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929496330398-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929496330398-392-0/minicluster-data/ts-0/wal
uuid: "3842c22109bd4100b90386d5469e8310"
format_stamp: "Formatted at 2026-06-20 04:28:58 on dist-test-slave-nk6d"
server_key: "cfbe2e829ac8a8944c0f2f7542738232"
server_key_iv: "ad88fb9b761de330a4fa11da3bec3466"
server_key_version: "encryptionkey@0"
I20260620 04:28:58.459931 19111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929496330398-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929496330398-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929496330398-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:28:58.483505 19111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:28:58.484239 19111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:28:58.484403 19111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:28:58.484990 19111 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:28:58.485922 19111 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:28:58.485967 19111 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:28:58.486008 19111 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:28:58.486035 19111 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:28:58.494246 19111 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37843
I20260620 04:28:58.494318 19238 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37843 every 8 connection(s)
I20260620 04:28:58.495142 19111 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929496330398-392-0/minicluster-data/ts-0/data/info.pb
I20260620 04:28:58.503568 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 19111
I20260620 04:28:58.503669 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929496330398-392-0/minicluster-data/ts-0/wal/instance
I20260620 04:28:58.503921 392 external_mini_cluster.cc:1428] Setting key e59404a8b0e282be6625055f6859a818
I20260620 04:28:58.504446 19090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:58.497072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47591 (local address 127.0.98.62:35035)
0620 04:28:58.497208 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 04:28:58.497212 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:28:58.498438 (+ 1226us) server_negotiation.cc:407] Connection header received
0620 04:28:58.499749 (+ 1311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:28:58.499752 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:28:58.499804 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:28:58.499881 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:28:58.500795 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:28:58.501323 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:28:58.502441 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:28:58.502676 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:28:58.502774 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:28:58.503913 (+ 1139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:28:58.503917 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:28:58.503921 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:28:58.503935 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:28:58.503983 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:28:58.503995 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:28:58.504056 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 04:28:58.504242 (+ 186us) server_negotiation.cc:299] Negotiation successful
0620 04:28:58.504319 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 04:28:58.505362 19241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:58.497557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35035 (local address 127.0.98.1:47591)
0620 04:28:58.498252 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0620 04:28:58.498292 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 04:28:58.499551 (+ 1259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:28:58.500071 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:28:58.500078 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:28:58.500163 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:28:58.500678 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:28:58.500690 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:28:58.501466 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:28:58.501469 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:28:58.502334 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:28:58.502343 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:28:58.502833 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:28:58.502844 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:28:58.503028 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:28:58.503639 (+ 611us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 04:28:58.503641 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:28:58.503698 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 04:28:58.503772 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:28:58.504092 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:28:58.504101 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:28:58.504107 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 04:28:58.504206 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 04:28:58.504501 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":129,"threads_started":1}
I20260620 04:28:58.505817 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929496330398-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929496330398-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929496330398-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929496330398-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35035
--builtin_ntp_servers=127.0.98.20:46365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 04:28:58.506682 19239 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35035
I20260620 04:28:58.506928 19239 heartbeater.cc:464] Registering TS with master...
I20260620 04:28:58.507457 19239 heartbeater.cc:511] Master 127.0.98.62:35035 requested a full tablet report, sending...
I20260620 04:28:58.509001 19051 ts_manager.cc:194] Registered new tserver with Master: 3842c22109bd4100b90386d5469e8310 (127.0.98.1:37843)
I20260620 04:28:58.510565 19051 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:47591
W20260620 04:28:58.612859 19242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:28:58.613127 19242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:28:58.613163 19242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:28:58.617122 19242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:28:58.617173 19242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:28:58.617225 19242 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 04:28:58.621876 19242 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929496330398-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929496330398-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929496330398-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35035
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19242
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929496330398-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:28:58.622946 19242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:28:58.624352 19242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:28:58.633567 19250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:28:58.633661 19248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:28:58.633554 19247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:28:58.633971 19242 server_base.cc:1034] running on GCE node
I20260620 04:28:58.634351 19242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:28:58.634968 19242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:28:58.636122 19242 hybrid_clock.cc:648] HybridClock initialized: now 1781929738636103 us; error 31 us; skew 500 ppm
I20260620 04:28:58.637835 19242 webserver.cc:460] Webserver started at http://127.0.98.2:46011/ using document root <none> and password file <none>
I20260620 04:28:58.638347 19242 fs_manager.cc:359] Metadata directory not provided
I20260620 04:28:58.638403 19242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:28:58.638572 19242 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:28:58.640573 19242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929496330398-392-0/minicluster-data/ts-1/data/instance:
uuid: "a6e80f7ab9084aa9b9d9df6f57c11466"
format_stamp: "Formatted at 2026-06-20 04:28:58 on dist-test-slave-nk6d"
server_key: "9c4a124759cbc0b9b754ea56bf93aa97"
server_key_iv: "a95702179c478e16de38794906d38275"
server_key_version: "encryptionkey@0"
I20260620 04:28:58.641062 19242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929496330398-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a6e80f7ab9084aa9b9d9df6f57c11466"
format_stamp: "Formatted at 2026-06-20 04:28:58 on dist-test-slave-nk6d"
server_key: "9c4a124759cbc0b9b754ea56bf93aa97"
server_key_iv: "a95702179c478e16de38794906d38275"
server_key_version: "encryptionkey@0"
I20260620 04:28:58.644642 19242 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:28:58.646915 19256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:28:58.648147 19242 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:28:58.648240 19242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929496330398-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929496330398-392-0/minicluster-data/ts-1/wal
uuid: "a6e80f7ab9084aa9b9d9df6f57c11466"
format_stamp: "Formatted at 2026-06-20 04:28:58 on dist-test-slave-nk6d"
server_key: "9c4a124759cbc0b9b754ea56bf93aa97"
server_key_iv: "a95702179c478e16de38794906d38275"
server_key_version: "encryptionkey@0"
I20260620 04:28:58.648339 19242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929496330398-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929496330398-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929496330398-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:28:58.670117 19242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:28:58.670794 19242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:28:58.670974 19242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:28:58.671567 19242 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:28:58.672541 19242 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:28:58.672600 19242 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:28:58.672641 19242 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:28:58.672669 19242 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:28:58.680904 19242 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42873
I20260620 04:28:58.680946 19369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42873 every 8 connection(s)
I20260620 04:28:58.681800 19242 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929496330398-392-0/minicluster-data/ts-1/data/info.pb
I20260620 04:28:58.681958 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 19242
I20260620 04:28:58.682078 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929496330398-392-0/minicluster-data/ts-1/wal/instance
I20260620 04:28:58.682390 392 external_mini_cluster.cc:1428] Setting key b660386d73e1ea939d7ec07c95b980bd
I20260620 04:28:58.684346 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929496330398-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929496330398-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929496330398-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929496330398-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35035
--builtin_ntp_servers=127.0.98.20:46365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 04:28:58.692170 19090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:58.683650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46467 (local address 127.0.98.62:35035)
0620 04:28:58.683782 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 04:28:58.683786 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:28:58.684988 (+ 1202us) server_negotiation.cc:407] Connection header received
0620 04:28:58.686627 (+ 1639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:28:58.686631 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:28:58.686676 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:28:58.686747 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:28:58.687952 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:28:58.688499 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:28:58.690204 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:28:58.690418 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:28:58.690507 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:28:58.691712 (+ 1205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:28:58.691716 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:28:58.691718 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:28:58.691730 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:28:58.691748 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:28:58.691756 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:28:58.691813 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 04:28:58.691998 (+ 185us) server_negotiation.cc:299] Negotiation successful
0620 04:28:58.692057 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 04:28:58.693454 19372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:58.684112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35035 (local address 127.0.98.2:46467)
0620 04:28:58.684829 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0620 04:28:58.684880 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 04:28:58.686385 (+ 1505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:28:58.687007 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:28:58.687017 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:28:58.687124 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:28:58.687784 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:28:58.687803 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:28:58.688619 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:28:58.688622 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:28:58.690077 (+ 1455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:28:58.690089 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:28:58.690568 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:28:58.690572 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:28:58.690772 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:28:58.691399 (+ 627us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 04:28:58.691410 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:28:58.691468 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 04:28:58.691553 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:28:58.691833 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:28:58.691841 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:28:58.691847 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 04:28:58.692012 (+ 165us) client_negotiation.cc:240] Negotiation successful
0620 04:28:58.692416 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":112,"threads_started":1}
I20260620 04:28:58.694585 19370 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35035
I20260620 04:28:58.694792 19370 heartbeater.cc:464] Registering TS with master...
I20260620 04:28:58.695262 19370 heartbeater.cc:511] Master 127.0.98.62:35035 requested a full tablet report, sending...
I20260620 04:28:58.696216 19051 ts_manager.cc:194] Registered new tserver with Master: a6e80f7ab9084aa9b9d9df6f57c11466 (127.0.98.2:42873)
I20260620 04:28:58.696731 19051 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:46467
W20260620 04:28:58.792675 19373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:28:58.792925 19373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:28:58.792977 19373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:28:58.796895 19373 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:28:58.796945 19373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:28:58.797015 19373 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 04:28:58.801653 19373 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929496330398-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929496330398-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929496330398-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35035
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19373
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929496330398-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:28:58.802632 19373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:28:58.803925 19373 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:28:58.812821 19381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:28:58.812815 19378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:28:58.812815 19379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:28:58.813161 19373 server_base.cc:1034] running on GCE node
I20260620 04:28:58.813498 19373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:28:58.814082 19373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:28:58.815227 19373 hybrid_clock.cc:648] HybridClock initialized: now 1781929738815207 us; error 29 us; skew 500 ppm
I20260620 04:28:58.816906 19373 webserver.cc:460] Webserver started at http://127.0.98.3:38753/ using document root <none> and password file <none>
I20260620 04:28:58.817476 19373 fs_manager.cc:359] Metadata directory not provided
I20260620 04:28:58.817535 19373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:28:58.817723 19373 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:28:58.819655 19373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929496330398-392-0/minicluster-data/ts-2/data/instance:
uuid: "cf48de98f011428b9b96858590e1f269"
format_stamp: "Formatted at 2026-06-20 04:28:58 on dist-test-slave-nk6d"
server_key: "1e56a3df96186975fd75ec7d858b0eb1"
server_key_iv: "f2b8c53f09fda3420459bd823cb3aabc"
server_key_version: "encryptionkey@0"
I20260620 04:28:58.820183 19373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929496330398-392-0/minicluster-data/ts-2/wal/instance:
uuid: "cf48de98f011428b9b96858590e1f269"
format_stamp: "Formatted at 2026-06-20 04:28:58 on dist-test-slave-nk6d"
server_key: "1e56a3df96186975fd75ec7d858b0eb1"
server_key_iv: "f2b8c53f09fda3420459bd823cb3aabc"
server_key_version: "encryptionkey@0"
I20260620 04:28:58.823518 19373 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260620 04:28:58.825850 19387 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:28:58.826992 19373 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 04:28:58.827083 19373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929496330398-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929496330398-392-0/minicluster-data/ts-2/wal
uuid: "cf48de98f011428b9b96858590e1f269"
format_stamp: "Formatted at 2026-06-20 04:28:58 on dist-test-slave-nk6d"
server_key: "1e56a3df96186975fd75ec7d858b0eb1"
server_key_iv: "f2b8c53f09fda3420459bd823cb3aabc"
server_key_version: "encryptionkey@0"
I20260620 04:28:58.827172 19373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929496330398-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929496330398-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929496330398-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:28:58.841094 19373 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:28:58.841684 19373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:28:58.841831 19373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:28:58.842350 19373 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:28:58.843212 19373 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:28:58.843256 19373 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:28:58.843295 19373 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:28:58.843322 19373 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:28:58.851423 19373 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42429
I20260620 04:28:58.851471 19500 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42429 every 8 connection(s)
I20260620 04:28:58.852322 19373 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929496330398-392-0/minicluster-data/ts-2/data/info.pb
I20260620 04:28:58.860392 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 19373
I20260620 04:28:58.860432 19090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:58.853963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54997 (local address 127.0.98.62:35035)
0620 04:28:58.854088 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 04:28:58.854092 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:28:58.855124 (+ 1032us) server_negotiation.cc:407] Connection header received
0620 04:28:58.856399 (+ 1275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:28:58.856401 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:28:58.856453 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:28:58.856524 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:28:58.857327 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:28:58.857849 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:28:58.858867 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:28:58.859080 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:28:58.859146 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:28:58.860044 (+ 898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:28:58.860048 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:28:58.860050 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:28:58.860062 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:28:58.860082 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:28:58.860090 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:28:58.860145 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 04:28:58.860273 (+ 128us) server_negotiation.cc:299] Negotiation successful
0620 04:28:58.860323 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 04:28:58.860518 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929496330398-392-0/minicluster-data/ts-2/wal/instance
I20260620 04:28:58.860795 392 external_mini_cluster.cc:1428] Setting key 347c89f5bc32435fd75fc657afa1249b
I20260620 04:28:58.861424 19503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:58.854345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35035 (local address 127.0.98.3:54997)
0620 04:28:58.854951 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0620 04:28:58.855011 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 04:28:58.856224 (+ 1213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:28:58.856663 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:28:58.856670 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:28:58.856748 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:28:58.857220 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:28:58.857232 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:28:58.857931 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:28:58.857933 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:28:58.858779 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:28:58.858787 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:28:58.859182 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:28:58.859185 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:28:58.859315 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:28:58.859796 (+ 481us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 04:28:58.859797 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:28:58.859833 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 04:28:58.859889 (+ 56us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:28:58.860151 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:28:58.860157 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:28:58.860162 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 04:28:58.860275 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 04:28:58.860537 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":113,"threads_started":1}
I20260620 04:28:58.862468 19501 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35035
I20260620 04:28:58.862682 19501 heartbeater.cc:464] Registering TS with master...
I20260620 04:28:58.863181 19501 heartbeater.cc:511] Master 127.0.98.62:35035 requested a full tablet report, sending...
I20260620 04:28:58.864109 19050 ts_manager.cc:194] Registered new tserver with Master: cf48de98f011428b9b96858590e1f269 (127.0.98.3:42429)
I20260620 04:28:58.864665 19050 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:54997
I20260620 04:28:58.874321 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 04:28:58.879525 19090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:58.876081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39410 (local address 127.0.98.62:35035)
0620 04:28:58.876199 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 04:28:58.876203 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:28:58.876350 (+ 147us) server_negotiation.cc:407] Connection header received
0620 04:28:58.876482 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:28:58.876485 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:28:58.876526 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:28:58.876590 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:28:58.876859 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:28:58.877380 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:28:58.878182 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:28:58.878396 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:28:58.878498 (+ 102us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:28:58.879158 (+ 660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:28:58.879161 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:28:58.879163 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:28:58.879174 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:28:58.879191 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:28:58.879198 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:28:58.879299 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 04:28:58.879381 (+ 82us) server_negotiation.cc:299] Negotiation successful
0620 04:28:58.879410 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 04:28:58.882120 19050 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39410:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 04:28:58.884939 19050 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 04:28:58.897954 19512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:58.892700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42873 (local address 127.0.0.1:35876)
0620 04:28:58.893433 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0620 04:28:58.893447 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:28:58.893528 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:28:58.894324 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:28:58.894331 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:28:58.894354 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:28:58.894603 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:28:58.894609 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:28:58.895545 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:28:58.895548 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:28:58.896509 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:28:58.896518 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:28:58.897441 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:28:58.897444 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:28:58.897563 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:28:58.897599 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 04:28:58.897699 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 04:28:58.897811 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":673,"mutex_wait_us":97,"thread_start_us":105,"threads_started":1}
I20260620 04:28:58.897959 19515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:58.893064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35876 (local address 127.0.98.2:42873)
0620 04:28:58.893830 (+ 766us) server_negotiation.cc:206] Beginning negotiation
0620 04:28:58.893847 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 04:28:58.893871 (+ 24us) server_negotiation.cc:407] Connection header received
0620 04:28:58.893948 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:28:58.893953 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:28:58.894162 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:28:58.894313 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:28:58.894722 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:28:58.895440 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:28:58.896618 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:28:58.897336 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:28:58.897427 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:28:58.897609 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0620 04:28:58.897733 (+ 124us) server_negotiation.cc:299] Negotiation successful
0620 04:28:58.897820 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":665,"thread_start_us":136,"threads_started":1}
I20260620 04:28:58.898394 19513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:58.892564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42429 (local address 127.0.0.1:32836)
0620 04:28:58.893192 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0620 04:28:58.893240 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 04:28:58.893363 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:28:58.894460 (+ 1097us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:28:58.894463 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:28:58.894475 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:28:58.894595 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:28:58.894601 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:28:58.895550 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:28:58.895554 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:28:58.896828 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:28:58.896838 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:28:58.898042 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:28:58.898045 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:28:58.898120 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:28:58.898142 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 04:28:58.898201 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 04:28:58.898261 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"mutex_wait_us":48,"thread_start_us":75,"threads_started":1}
I20260620 04:28:58.899274 19516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:58.893269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37843 (local address 127.0.0.1:35516)
0620 04:28:58.893703 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0620 04:28:58.893714 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 04:28:58.893787 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:28:58.894352 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:28:58.894355 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:28:58.894367 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:28:58.894591 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:28:58.894600 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:28:58.896878 (+ 2278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:28:58.896882 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:28:58.897951 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:28:58.897972 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:28:58.898994 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:28:58.898998 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:28:58.899069 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:28:58.899087 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 04:28:58.899131 (+ 44us) client_negotiation.cc:240] Negotiation successful
0620 04:28:58.899179 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"mutex_wait_us":46,"spinlock_wait_cycles":2304,"thread_start_us":81,"threads_started":1}
I20260620 04:28:58.899410 19517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:58.893651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35516 (local address 127.0.98.1:37843)
0620 04:28:58.894000 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0620 04:28:58.894010 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 04:28:58.894028 (+ 18us) server_negotiation.cc:407] Connection header received
0620 04:28:58.894085 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:28:58.894089 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:28:58.894223 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:28:58.894536 (+ 313us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:28:58.895577 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:28:58.896753 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:28:58.898105 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:28:58.898856 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:28:58.898967 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:28:58.899139 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0620 04:28:58.899184 (+ 45us) server_negotiation.cc:299] Negotiation successful
0620 04:28:58.899265 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":102,"threads_started":1}
I20260620 04:28:58.898912 19514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:58.892701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32836 (local address 127.0.98.3:42429)
0620 04:28:58.893891 (+ 1190us) server_negotiation.cc:206] Beginning negotiation
0620 04:28:58.893904 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 04:28:58.893924 (+ 20us) server_negotiation.cc:407] Connection header received
0620 04:28:58.893984 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:28:58.893989 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:28:58.894162 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:28:58.894313 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:28:58.894707 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:28:58.895441 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:28:58.896949 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:28:58.897728 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:28:58.897822 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:28:58.897984 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 04:28:58.898655 (+ 671us) server_negotiation.cc:299] Negotiation successful
0620 04:28:58.898752 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":838,"thread_start_us":98,"threads_started":1}
I20260620 04:28:58.901283 19304 tablet_service.cc:1467] Processing CreateTablet for tablet da23d37f925741a3b9bd0e15b0f4eeb1 (DEFAULT_TABLE table=test-table [id=6a92deb8ff3d4d288e79276f5d47cf69]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:28:58.901724 19173 tablet_service.cc:1467] Processing CreateTablet for tablet da23d37f925741a3b9bd0e15b0f4eeb1 (DEFAULT_TABLE table=test-table [id=6a92deb8ff3d4d288e79276f5d47cf69]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:28:58.902381 19304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da23d37f925741a3b9bd0e15b0f4eeb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:28:58.902567 19173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da23d37f925741a3b9bd0e15b0f4eeb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:28:58.902231 19435 tablet_service.cc:1467] Processing CreateTablet for tablet da23d37f925741a3b9bd0e15b0f4eeb1 (DEFAULT_TABLE table=test-table [id=6a92deb8ff3d4d288e79276f5d47cf69]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:28:58.903152 19435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da23d37f925741a3b9bd0e15b0f4eeb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:28:58.909029 19518 tablet_bootstrap.cc:492] T da23d37f925741a3b9bd0e15b0f4eeb1 P 3842c22109bd4100b90386d5469e8310: Bootstrap starting.
I20260620 04:28:58.909816 19520 tablet_bootstrap.cc:492] T da23d37f925741a3b9bd0e15b0f4eeb1 P cf48de98f011428b9b96858590e1f269: Bootstrap starting.
I20260620 04:28:58.910310 19519 tablet_bootstrap.cc:492] T da23d37f925741a3b9bd0e15b0f4eeb1 P a6e80f7ab9084aa9b9d9df6f57c11466: Bootstrap starting.
I20260620 04:28:58.911347 19518 tablet_bootstrap.cc:654] T da23d37f925741a3b9bd0e15b0f4eeb1 P 3842c22109bd4100b90386d5469e8310: Neither blocks nor log segments found. Creating new log.
I20260620 04:28:58.911890 19519 tablet_bootstrap.cc:654] T da23d37f925741a3b9bd0e15b0f4eeb1 P a6e80f7ab9084aa9b9d9df6f57c11466: Neither blocks nor log segments found. Creating new log.
I20260620 04:28:58.911944 19520 tablet_bootstrap.cc:654] T da23d37f925741a3b9bd0e15b0f4eeb1 P cf48de98f011428b9b96858590e1f269: Neither blocks nor log segments found. Creating new log.
I20260620 04:28:58.912231 19518 log.cc:824] T da23d37f925741a3b9bd0e15b0f4eeb1 P 3842c22109bd4100b90386d5469e8310: Log is configured to *not* fsync() on all Append() calls
I20260620 04:28:58.912570 19519 log.cc:824] T da23d37f925741a3b9bd0e15b0f4eeb1 P a6e80f7ab9084aa9b9d9df6f57c11466: Log is configured to *not* fsync() on all Append() calls
I20260620 04:28:58.912739 19520 log.cc:824] T da23d37f925741a3b9bd0e15b0f4eeb1 P cf48de98f011428b9b96858590e1f269: Log is configured to *not* fsync() on all Append() calls
I20260620 04:28:58.914191 19518 tablet_bootstrap.cc:492] T da23d37f925741a3b9bd0e15b0f4eeb1 P 3842c22109bd4100b90386d5469e8310: No bootstrap required, opened a new log
I20260620 04:28:58.914355 19518 ts_tablet_manager.cc:1397] T da23d37f925741a3b9bd0e15b0f4eeb1 P 3842c22109bd4100b90386d5469e8310: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260620 04:28:58.914414 19519 tablet_bootstrap.cc:492] T da23d37f925741a3b9bd0e15b0f4eeb1 P a6e80f7ab9084aa9b9d9df6f57c11466: No bootstrap required, opened a new log
I20260620 04:28:58.914583 19519 ts_tablet_manager.cc:1397] T da23d37f925741a3b9bd0e15b0f4eeb1 P a6e80f7ab9084aa9b9d9df6f57c11466: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260620 04:28:58.915023 19520 tablet_bootstrap.cc:492] T da23d37f925741a3b9bd0e15b0f4eeb1 P cf48de98f011428b9b96858590e1f269: No bootstrap required, opened a new log
I20260620 04:28:58.915195 19520 ts_tablet_manager.cc:1397] T da23d37f925741a3b9bd0e15b0f4eeb1 P cf48de98f011428b9b96858590e1f269: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260620 04:28:58.917547 19518 raft_consensus.cc:348] T da23d37f925741a3b9bd0e15b0f4eeb1 P 3842c22109bd4100b90386d5469e8310 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf48de98f011428b9b96858590e1f269" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42429 } } peers { permanent_uuid: "a6e80f7ab9084aa9b9d9df6f57c11466" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42873 } } peers { permanent_uuid: "3842c22109bd4100b90386d5469e8310" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37843 } }
I20260620 04:28:58.917766 19518 raft_consensus.cc:374] T da23d37f925741a3b9bd0e15b0f4eeb1 P 3842c22109bd4100b90386d5469e8310 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:28:58.917819 19518 raft_consensus.cc:729] T da23d37f925741a3b9bd0e15b0f4eeb1 P 3842c22109bd4100b90386d5469e8310 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3842c22109bd4100b90386d5469e8310, State: Initialized, Role: FOLLOWER
I20260620 04:28:58.918213 19518 consensus_queue.cc:260] T da23d37f925741a3b9bd0e15b0f4eeb1 P 3842c22109bd4100b90386d5469e8310 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf48de98f011428b9b96858590e1f269" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42429 } } peers { permanent_uuid: "a6e80f7ab9084aa9b9d9df6f57c11466" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42873 } } peers { permanent_uuid: "3842c22109bd4100b90386d5469e8310" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37843 } }
I20260620 04:28:58.918787 19239 heartbeater.cc:503] Master 127.0.98.62:35035 was elected leader, sending a full tablet report...
I20260620 04:28:58.918648 19519 raft_consensus.cc:348] T da23d37f925741a3b9bd0e15b0f4eeb1 P a6e80f7ab9084aa9b9d9df6f57c11466 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf48de98f011428b9b96858590e1f269" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42429 } } peers { permanent_uuid: "a6e80f7ab9084aa9b9d9df6f57c11466" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42873 } } peers { permanent_uuid: "3842c22109bd4100b90386d5469e8310" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37843 } }
I20260620 04:28:58.918898 19519 raft_consensus.cc:374] T da23d37f925741a3b9bd0e15b0f4eeb1 P a6e80f7ab9084aa9b9d9df6f57c11466 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:28:58.918956 19519 raft_consensus.cc:729] T da23d37f925741a3b9bd0e15b0f4eeb1 P a6e80f7ab9084aa9b9d9df6f57c11466 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e80f7ab9084aa9b9d9df6f57c11466, State: Initialized, Role: FOLLOWER
I20260620 04:28:58.919098 19520 raft_consensus.cc:348] T da23d37f925741a3b9bd0e15b0f4eeb1 P cf48de98f011428b9b96858590e1f269 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf48de98f011428b9b96858590e1f269" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42429 } } peers { permanent_uuid: "a6e80f7ab9084aa9b9d9df6f57c11466" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42873 } } peers { permanent_uuid: "3842c22109bd4100b90386d5469e8310" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37843 } }
I20260620 04:28:58.919323 19520 raft_consensus.cc:374] T da23d37f925741a3b9bd0e15b0f4eeb1 P cf48de98f011428b9b96858590e1f269 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:28:58.919371 19520 raft_consensus.cc:729] T da23d37f925741a3b9bd0e15b0f4eeb1 P cf48de98f011428b9b96858590e1f269 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf48de98f011428b9b96858590e1f269, State: Initialized, Role: FOLLOWER
I20260620 04:28:58.919375 19519 consensus_queue.cc:260] T da23d37f925741a3b9bd0e15b0f4eeb1 P a6e80f7ab9084aa9b9d9df6f57c11466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf48de98f011428b9b96858590e1f269" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42429 } } peers { permanent_uuid: "a6e80f7ab9084aa9b9d9df6f57c11466" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42873 } } peers { permanent_uuid: "3842c22109bd4100b90386d5469e8310" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37843 } }
I20260620 04:28:58.919760 19520 consensus_queue.cc:260] T da23d37f925741a3b9bd0e15b0f4eeb1 P cf48de98f011428b9b96858590e1f269 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf48de98f011428b9b96858590e1f269" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42429 } } peers { permanent_uuid: "a6e80f7ab9084aa9b9d9df6f57c11466" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42873 } } peers { permanent_uuid: "3842c22109bd4100b90386d5469e8310" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37843 } }
I20260620 04:28:58.920154 19518 ts_tablet_manager.cc:1428] T da23d37f925741a3b9bd0e15b0f4eeb1 P 3842c22109bd4100b90386d5469e8310: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260620 04:28:58.920297 19519 ts_tablet_manager.cc:1428] T da23d37f925741a3b9bd0e15b0f4eeb1 P a6e80f7ab9084aa9b9d9df6f57c11466: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 04:28:58.920620 19370 heartbeater.cc:503] Master 127.0.98.62:35035 was elected leader, sending a full tablet report...
I20260620 04:28:58.922116 19501 heartbeater.cc:503] Master 127.0.98.62:35035 was elected leader, sending a full tablet report...
I20260620 04:28:58.923184 19520 ts_tablet_manager.cc:1428] T da23d37f925741a3b9bd0e15b0f4eeb1 P cf48de98f011428b9b96858590e1f269: Time spent starting tablet: real 0.008s user 0.004s sys 0.001s
W20260620 04:28:58.932809 19371 tablet.cc:2369] T da23d37f925741a3b9bd0e15b0f4eeb1 P a6e80f7ab9084aa9b9d9df6f57c11466: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 04:28:58.996191 19240 tablet.cc:2369] T da23d37f925741a3b9bd0e15b0f4eeb1 P 3842c22109bd4100b90386d5469e8310: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 04:28:59.103246 19502 tablet.cc:2369] T da23d37f925741a3b9bd0e15b0f4eeb1 P cf48de98f011428b9b96858590e1f269: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 04:28:59.329581 19524 raft_consensus.cc:482] T da23d37f925741a3b9bd0e15b0f4eeb1 P 3842c22109bd4100b90386d5469e8310 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:28:59.329808 19524 raft_consensus.cc:504] T da23d37f925741a3b9bd0e15b0f4eeb1 P 3842c22109bd4100b90386d5469e8310 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf48de98f011428b9b96858590e1f269" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42429 } } peers { permanent_uuid: "a6e80f7ab9084aa9b9d9df6f57c11466" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42873 } } peers { permanent_uuid: "3842c22109bd4100b90386d5469e8310" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37843 } }
I20260620 04:28:59.331183 19524 leader_election.cc:288] T da23d37f925741a3b9bd0e15b0f4eeb1 P 3842c22109bd4100b90386d5469e8310 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cf48de98f011428b9b96858590e1f269 (127.0.98.3:42429), a6e80f7ab9084aa9b9d9df6f57c11466 (127.0.98.2:42873)
I20260620 04:28:59.335078 19515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:59.331645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34057 (local address 127.0.98.2:42873)
0620 04:28:59.331786 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 04:28:59.331791 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:28:59.332030 (+ 239us) server_negotiation.cc:407] Connection header received
0620 04:28:59.332208 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:28:59.332211 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:28:59.332254 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:28:59.332315 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:28:59.332588 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:28:59.333083 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:28:59.334213 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:28:59.334582 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:28:59.334672 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:28:59.334730 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 04:28:59.334896 (+ 166us) server_negotiation.cc:299] Negotiation successful
0620 04:28:59.334961 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 04:28:59.335098 19528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:59.331478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42873 (local address 127.0.98.1:34057)
0620 04:28:59.331929 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0620 04:28:59.331940 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 04:28:59.332097 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:28:59.332331 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:28:59.332334 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:28:59.332356 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:28:59.332485 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:28:59.332490 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:28:59.333178 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:28:59.333181 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:28:59.334096 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:28:59.334105 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:28:59.334688 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:28:59.334691 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:28:59.334783 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:28:59.334806 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 04:28:59.334870 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 04:28:59.334938 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":390,"mutex_wait_us":72,"thread_start_us":84,"threads_started":1}
I20260620 04:28:59.335204 19514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:59.331645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45027 (local address 127.0.98.3:42429)
0620 04:28:59.331793 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 04:28:59.331798 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:28:59.331976 (+ 178us) server_negotiation.cc:407] Connection header received
0620 04:28:59.332081 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:28:59.332085 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:28:59.332135 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:28:59.332202 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:28:59.332504 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:28:59.333049 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:28:59.334225 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:28:59.334635 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:28:59.334737 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:28:59.334801 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 04:28:59.334985 (+ 184us) server_negotiation.cc:299] Negotiation successful
0620 04:28:59.335060 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 04:28:59.335418 19527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:59.331478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42429 (local address 127.0.98.1:45027)
0620 04:28:59.331804 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0620 04:28:59.331819 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 04:28:59.331951 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:28:59.332224 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:28:59.332227 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:28:59.332244 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:28:59.332372 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:28:59.332378 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:28:59.333167 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:28:59.333171 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:28:59.334101 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:28:59.334111 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:28:59.334766 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:28:59.334770 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:28:59.334861 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:28:59.334881 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 04:28:59.334956 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 04:28:59.335030 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"mutex_wait_us":32,"thread_start_us":81,"threads_started":1}
I20260620 04:28:59.335855 19324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da23d37f925741a3b9bd0e15b0f4eeb1" candidate_uuid: "3842c22109bd4100b90386d5469e8310" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e80f7ab9084aa9b9d9df6f57c11466" is_pre_election: true
I20260620 04:28:59.335943 19455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da23d37f925741a3b9bd0e15b0f4eeb1" candidate_uuid: "3842c22109bd4100b90386d5469e8310" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf48de98f011428b9b96858590e1f269" is_pre_election: true
I20260620 04:28:59.336215 19324 raft_consensus.cc:2454] T da23d37f925741a3b9bd0e15b0f4eeb1 P a6e80f7ab9084aa9b9d9df6f57c11466 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3842c22109bd4100b90386d5469e8310 in term 0.
I20260620 04:28:59.336215 19455 raft_consensus.cc:2454] T da23d37f925741a3b9bd0e15b0f4eeb1 P cf48de98f011428b9b96858590e1f269 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3842c22109bd4100b90386d5469e8310 in term 0.
I20260620 04:28:59.336656 19128 leader_election.cc:302] T da23d37f925741a3b9bd0e15b0f4eeb1 P 3842c22109bd4100b90386d5469e8310 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3842c22109bd4100b90386d5469e8310, cf48de98f011428b9b96858590e1f269; no voters:
I20260620 04:28:59.336894 19524 raft_consensus.cc:2781] T da23d37f925741a3b9bd0e15b0f4eeb1 P 3842c22109bd4100b90386d5469e8310 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 04:28:59.336985 19524 raft_consensus.cc:482] T da23d37f925741a3b9bd0e15b0f4eeb1 P 3842c22109bd4100b90386d5469e8310 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 04:28:59.337036 19524 raft_consensus.cc:3037] T da23d37f925741a3b9bd0e15b0f4eeb1 P 3842c22109bd4100b90386d5469e8310 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:28:59.338009 19524 raft_consensus.cc:504] T da23d37f925741a3b9bd0e15b0f4eeb1 P 3842c22109bd4100b90386d5469e8310 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf48de98f011428b9b96858590e1f269" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42429 } } peers { permanent_uuid: "a6e80f7ab9084aa9b9d9df6f57c11466" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42873 } } peers { permanent_uuid: "3842c22109bd4100b90386d5469e8310" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37843 } }
I20260620 04:28:59.338321 19524 leader_election.cc:288] T da23d37f925741a3b9bd0e15b0f4eeb1 P 3842c22109bd4100b90386d5469e8310 [CANDIDATE]: Term 1 election: Requested vote from peers cf48de98f011428b9b96858590e1f269 (127.0.98.3:42429), a6e80f7ab9084aa9b9d9df6f57c11466 (127.0.98.2:42873)
I20260620 04:28:59.338693 19455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da23d37f925741a3b9bd0e15b0f4eeb1" candidate_uuid: "3842c22109bd4100b90386d5469e8310" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf48de98f011428b9b96858590e1f269"
I20260620 04:28:59.338683 19324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da23d37f925741a3b9bd0e15b0f4eeb1" candidate_uuid: "3842c22109bd4100b90386d5469e8310" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e80f7ab9084aa9b9d9df6f57c11466"
I20260620 04:28:59.338820 19324 raft_consensus.cc:3037] T da23d37f925741a3b9bd0e15b0f4eeb1 P a6e80f7ab9084aa9b9d9df6f57c11466 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:28:59.338820 19455 raft_consensus.cc:3037] T da23d37f925741a3b9bd0e15b0f4eeb1 P cf48de98f011428b9b96858590e1f269 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:28:59.340131 19324 raft_consensus.cc:2454] T da23d37f925741a3b9bd0e15b0f4eeb1 P a6e80f7ab9084aa9b9d9df6f57c11466 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3842c22109bd4100b90386d5469e8310 in term 1.
I20260620 04:28:59.340131 19455 raft_consensus.cc:2454] T da23d37f925741a3b9bd0e15b0f4eeb1 P cf48de98f011428b9b96858590e1f269 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3842c22109bd4100b90386d5469e8310 in term 1.
I20260620 04:28:59.340445 19128 leader_election.cc:302] T da23d37f925741a3b9bd0e15b0f4eeb1 P 3842c22109bd4100b90386d5469e8310 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3842c22109bd4100b90386d5469e8310, cf48de98f011428b9b96858590e1f269; no voters:
I20260620 04:28:59.340658 19524 raft_consensus.cc:2781] T da23d37f925741a3b9bd0e15b0f4eeb1 P 3842c22109bd4100b90386d5469e8310 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:28:59.340854 19524 raft_consensus.cc:686] T da23d37f925741a3b9bd0e15b0f4eeb1 P 3842c22109bd4100b90386d5469e8310 [term 1 LEADER]: Becoming Leader. State: Replica: 3842c22109bd4100b90386d5469e8310, State: Running, Role: LEADER
I20260620 04:28:59.341203 19524 consensus_queue.cc:237] T da23d37f925741a3b9bd0e15b0f4eeb1 P 3842c22109bd4100b90386d5469e8310 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf48de98f011428b9b96858590e1f269" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42429 } } peers { permanent_uuid: "a6e80f7ab9084aa9b9d9df6f57c11466" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42873 } } peers { permanent_uuid: "3842c22109bd4100b90386d5469e8310" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37843 } }
I20260620 04:28:59.344282 19050 catalog_manager.cc:5495] T da23d37f925741a3b9bd0e15b0f4eeb1 P 3842c22109bd4100b90386d5469e8310 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3842c22109bd4100b90386d5469e8310 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "3842c22109bd4100b90386d5469e8310" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf48de98f011428b9b96858590e1f269" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42429 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6e80f7ab9084aa9b9d9df6f57c11466" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3842c22109bd4100b90386d5469e8310" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37843 } health_report { overall_health: HEALTHY } } }
I20260620 04:28:59.459321 19533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:59.455651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35526 (local address 127.0.98.1:37843)
0620 04:28:59.455879 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0620 04:28:59.455883 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:28:59.455898 (+ 15us) server_negotiation.cc:407] Connection header received
0620 04:28:59.456096 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:28:59.456099 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:28:59.456158 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:28:59.456252 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:28:59.456493 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:28:59.457009 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:28:59.457840 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:28:59.458073 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:28:59.458157 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:28:59.458475 (+ 318us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:28:59.458903 (+ 428us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:28:59.459035 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 04:28:59.459137 (+ 102us) server_negotiation.cc:299] Negotiation successful
0620 04:28:59.459169 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":59,"threads_started":1}
I20260620 04:28:59.465263 19455 raft_consensus.cc:1261] T da23d37f925741a3b9bd0e15b0f4eeb1 P cf48de98f011428b9b96858590e1f269 [term 1 FOLLOWER]: Refusing update from remote peer 3842c22109bd4100b90386d5469e8310: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:28:59.465309 19324 raft_consensus.cc:1261] T da23d37f925741a3b9bd0e15b0f4eeb1 P a6e80f7ab9084aa9b9d9df6f57c11466 [term 1 FOLLOWER]: Refusing update from remote peer 3842c22109bd4100b90386d5469e8310: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:28:59.465961 19529 consensus_queue.cc:1035] T da23d37f925741a3b9bd0e15b0f4eeb1 P 3842c22109bd4100b90386d5469e8310 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf48de98f011428b9b96858590e1f269" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:28:59.466131 19524 consensus_queue.cc:1035] T da23d37f925741a3b9bd0e15b0f4eeb1 P 3842c22109bd4100b90386d5469e8310 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e80f7ab9084aa9b9d9df6f57c11466" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:28:59.474893 19534 mvcc.cc:205] Tried to move back new op lower bound from 7298784212841828352 to 7298784212344139776. Current Snapshot: MvccSnapshot[applied={T|T < 7298784212841828352}]
I20260620 04:28:59.477593 19537 mvcc.cc:205] Tried to move back new op lower bound from 7298784212841828352 to 7298784212344139776. Current Snapshot: MvccSnapshot[applied={T|T < 7298784212841828352 or (T in {7298784212841828352})}]
I20260620 04:28:59.482826 19050 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39410:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:28:59.482973 19050 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39410:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:28:59.485145 19050 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 75acd78b79304fe08d8d0d04e816de34: Sending DeleteTablet for 3 replicas of tablet da23d37f925741a3b9bd0e15b0f4eeb1
I20260620 04:28:59.485833 19304 tablet_service.cc:1514] Processing DeleteTablet for tablet da23d37f925741a3b9bd0e15b0f4eeb1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:28:59 UTC) from {username='slave'} at 127.0.0.1:35876
I20260620 04:28:59.485890 19435 tablet_service.cc:1514] Processing DeleteTablet for tablet da23d37f925741a3b9bd0e15b0f4eeb1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:28:59 UTC) from {username='slave'} at 127.0.0.1:32836
I20260620 04:28:59.486274 19546 tablet_replica.cc:330] T da23d37f925741a3b9bd0e15b0f4eeb1 P a6e80f7ab9084aa9b9d9df6f57c11466: stopping tablet replica
I20260620 04:28:59.486583 19546 raft_consensus.cc:2229] T da23d37f925741a3b9bd0e15b0f4eeb1 P a6e80f7ab9084aa9b9d9df6f57c11466 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:28:59.486559 19173 tablet_service.cc:1514] Processing DeleteTablet for tablet da23d37f925741a3b9bd0e15b0f4eeb1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:28:59 UTC) from {username='slave'} at 127.0.0.1:35516
I20260620 04:28:59.486757 19547 tablet_replica.cc:330] T da23d37f925741a3b9bd0e15b0f4eeb1 P cf48de98f011428b9b96858590e1f269: stopping tablet replica
I20260620 04:28:59.486891 19546 raft_consensus.cc:2258] T da23d37f925741a3b9bd0e15b0f4eeb1 P a6e80f7ab9084aa9b9d9df6f57c11466 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:28:59.487015 19547 raft_consensus.cc:2229] T da23d37f925741a3b9bd0e15b0f4eeb1 P cf48de98f011428b9b96858590e1f269 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:28:59.487056 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 19111
I20260620 04:28:59.487977 19547 raft_consensus.cc:2258] T da23d37f925741a3b9bd0e15b0f4eeb1 P cf48de98f011428b9b96858590e1f269 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:28:59.489212 19546 ts_tablet_manager.cc:1905] T da23d37f925741a3b9bd0e15b0f4eeb1 P a6e80f7ab9084aa9b9d9df6f57c11466: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:28:59.489212 19547 ts_tablet_manager.cc:1905] T da23d37f925741a3b9bd0e15b0f4eeb1 P cf48de98f011428b9b96858590e1f269: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:28:59.491782 19546 ts_tablet_manager.cc:1918] T da23d37f925741a3b9bd0e15b0f4eeb1 P a6e80f7ab9084aa9b9d9df6f57c11466: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:28:59.491781 19547 ts_tablet_manager.cc:1918] T da23d37f925741a3b9bd0e15b0f4eeb1 P cf48de98f011428b9b96858590e1f269: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:28:59.491880 19547 log.cc:1196] T da23d37f925741a3b9bd0e15b0f4eeb1 P cf48de98f011428b9b96858590e1f269: Deleting WAL directory at /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929496330398-392-0/minicluster-data/ts-2/wal/wals/da23d37f925741a3b9bd0e15b0f4eeb1
I20260620 04:28:59.491880 19546 log.cc:1196] T da23d37f925741a3b9bd0e15b0f4eeb1 P a6e80f7ab9084aa9b9d9df6f57c11466: Deleting WAL directory at /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929496330398-392-0/minicluster-data/ts-1/wal/wals/da23d37f925741a3b9bd0e15b0f4eeb1
I20260620 04:28:59.492202 19546 ts_tablet_manager.cc:1939] T da23d37f925741a3b9bd0e15b0f4eeb1 P a6e80f7ab9084aa9b9d9df6f57c11466: Deleting consensus metadata
I20260620 04:28:59.492202 19547 ts_tablet_manager.cc:1939] T da23d37f925741a3b9bd0e15b0f4eeb1 P cf48de98f011428b9b96858590e1f269: Deleting consensus metadata
I20260620 04:28:59.493392 19035 catalog_manager.cc:4841] TS a6e80f7ab9084aa9b9d9df6f57c11466 (127.0.98.2:42873): tablet da23d37f925741a3b9bd0e15b0f4eeb1 (table test-table [id=6a92deb8ff3d4d288e79276f5d47cf69]) successfully deleted
I20260620 04:28:59.493456 19037 catalog_manager.cc:4841] TS cf48de98f011428b9b96858590e1f269 (127.0.98.3:42429): tablet da23d37f925741a3b9bd0e15b0f4eeb1 (table test-table [id=6a92deb8ff3d4d288e79276f5d47cf69]) successfully deleted
W20260620 04:28:59.493685 19037 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37843 (error 108)
I20260620 04:28:59.493827 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 19242
I20260620 04:28:59.495172 19550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:28:59.494752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37843 (local address 127.0.0.1:35538)
0620 04:28:59.495007 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0620 04:28:59.495078 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":64,"threads_started":1}
W20260620 04:28:59.495293 19037 catalog_manager.cc:4584] TS 3842c22109bd4100b90386d5469e8310 (127.0.98.1:37843): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet da23d37f925741a3b9bd0e15b0f4eeb1: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37843: connect: Connection refused (error 111)
I20260620 04:28:59.498855 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 19373
I20260620 04:28:59.503578 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 19020
2026-06-20T04:28:59Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1445 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929496330398-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:28:59 dist-test-slave-nk6d krb5kdc[19553](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:28:59 dist-test-slave-nk6d krb5kdc[19553](info): set up 2 sockets
Jun 20 04:28:59 dist-test-slave-nk6d krb5kdc[19553](info): commencing operation
krb5kdc: starting...
W20260620 04:29:00.447649 392 mini_kdc.cc:121] Time spent starting KDC: real 0.924s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929496330398-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929496330398-392-0/krb5kdc/test-user.keytab.
Jun 20 04:29:00 dist-test-slave-nk6d krb5kdc[19553](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929740, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:29:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:29:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 04:29:00.558285 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929496330398-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929496330398-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929496330398-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37061
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37061
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:29:00.665655 19566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:29:00.665930 19566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:29:00.665995 19566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:29:00.670214 19566 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:29:00.670338 19566 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:29:00.670405 19566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:29:00.670439 19566 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:29:00.670468 19566 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:29:00.675702 19566 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929496330398-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929496330398-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37061
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19566
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929496330398-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:29:00.676683 19566 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:29:00.677659 19566 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:29:00.685992 19571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:29:00.686038 19574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:29:00.686108 19572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:29:00.686314 19566 server_base.cc:1034] running on GCE node
I20260620 04:29:00.686882 19566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:29:00.688139 19566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:29:00.689304 19566 hybrid_clock.cc:648] HybridClock initialized: now 1781929740689276 us; error 40 us; skew 500 ppm
Jun 20 04:29:00 dist-test-slave-nk6d krb5kdc[19553](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929740, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:29:00.691906 19566 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 04:29:00.692838 19566 webserver.cc:460] Webserver started at http://127.0.98.62:34757/ using document root <none> and password file <none>
I20260620 04:29:00.693364 19566 fs_manager.cc:359] Metadata directory not provided
I20260620 04:29:00.693424 19566 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:29:00.693604 19566 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:29:00.695554 19566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929496330398-392-0/minicluster-data/master-0/data/instance:
uuid: "25c6adebd0e74ef28f3851dfd7db22d8"
format_stamp: "Formatted at 2026-06-20 04:29:00 on dist-test-slave-nk6d"
server_key: "e46e9e71321084fdf401fecdedcc1d1a"
server_key_iv: "d19535bc5d66f6a8a7bd418a792be87e"
server_key_version: "encryptionkey@0"
I20260620 04:29:00.696102 19566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929496330398-392-0/minicluster-data/master-0/wal/instance:
uuid: "25c6adebd0e74ef28f3851dfd7db22d8"
format_stamp: "Formatted at 2026-06-20 04:29:00 on dist-test-slave-nk6d"
server_key: "e46e9e71321084fdf401fecdedcc1d1a"
server_key_iv: "d19535bc5d66f6a8a7bd418a792be87e"
server_key_version: "encryptionkey@0"
I20260620 04:29:00.699536 19566 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 04:29:00.701745 19581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:29:00.703033 19566 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 04:29:00.703181 19566 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929496330398-392-0/minicluster-data/master-0/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929496330398-392-0/minicluster-data/master-0/wal
uuid: "25c6adebd0e74ef28f3851dfd7db22d8"
format_stamp: "Formatted at 2026-06-20 04:29:00 on dist-test-slave-nk6d"
server_key: "e46e9e71321084fdf401fecdedcc1d1a"
server_key_iv: "d19535bc5d66f6a8a7bd418a792be87e"
server_key_version: "encryptionkey@0"
I20260620 04:29:00.703289 19566 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929496330398-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929496330398-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929496330398-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:29:00.713356 19566 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:29:00.715925 19566 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:29:00.716109 19566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:29:00.722576 19566 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37061
I20260620 04:29:00.722645 19633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37061 every 8 connection(s)
I20260620 04:29:00.723474 19566 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:29:00.726296 19634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:29:00.731659 19634 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25c6adebd0e74ef28f3851dfd7db22d8: Bootstrap starting.
I20260620 04:29:00.733745 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 19566
I20260620 04:29:00.733884 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929496330398-392-0/minicluster-data/master-0/wal/instance
I20260620 04:29:00.734179 392 external_mini_cluster.cc:1428] Setting key ce44b45b183aaed7de2bd4e7c7e63730
I20260620 04:29:00.734376 19634 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 25c6adebd0e74ef28f3851dfd7db22d8: Neither blocks nor log segments found. Creating new log.
I20260620 04:29:00.735186 19634 log.cc:824] T 00000000000000000000000000000000 P 25c6adebd0e74ef28f3851dfd7db22d8: Log is configured to *not* fsync() on all Append() calls
I20260620 04:29:00.737241 19634 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25c6adebd0e74ef28f3851dfd7db22d8: No bootstrap required, opened a new log
I20260620 04:29:00.740629 19634 raft_consensus.cc:348] T 00000000000000000000000000000000 P 25c6adebd0e74ef28f3851dfd7db22d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25c6adebd0e74ef28f3851dfd7db22d8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37061 } }
I20260620 04:29:00.740844 19634 raft_consensus.cc:374] T 00000000000000000000000000000000 P 25c6adebd0e74ef28f3851dfd7db22d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:29:00.740912 19634 raft_consensus.cc:729] T 00000000000000000000000000000000 P 25c6adebd0e74ef28f3851dfd7db22d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25c6adebd0e74ef28f3851dfd7db22d8, State: Initialized, Role: FOLLOWER
I20260620 04:29:00.741313 19634 consensus_queue.cc:260] T 00000000000000000000000000000000 P 25c6adebd0e74ef28f3851dfd7db22d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25c6adebd0e74ef28f3851dfd7db22d8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37061 } }
I20260620 04:29:00.741441 19634 raft_consensus.cc:388] T 00000000000000000000000000000000 P 25c6adebd0e74ef28f3851dfd7db22d8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:29:00.741503 19634 raft_consensus.cc:482] T 00000000000000000000000000000000 P 25c6adebd0e74ef28f3851dfd7db22d8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:29:00.741600 19634 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 25c6adebd0e74ef28f3851dfd7db22d8 [term 0 FOLLOWER]: Advancing to term 1
Jun 20 04:29:00 dist-test-slave-nk6d krb5kdc[19553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929740, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:29:00.742660 19634 raft_consensus.cc:504] T 00000000000000000000000000000000 P 25c6adebd0e74ef28f3851dfd7db22d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25c6adebd0e74ef28f3851dfd7db22d8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37061 } }
I20260620 04:29:00.742977 19634 leader_election.cc:302] T 00000000000000000000000000000000 P 25c6adebd0e74ef28f3851dfd7db22d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 25c6adebd0e74ef28f3851dfd7db22d8; no voters:
I20260620 04:29:00.743289 19634 leader_election.cc:288] T 00000000000000000000000000000000 P 25c6adebd0e74ef28f3851dfd7db22d8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:29:00.743407 19639 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 25c6adebd0e74ef28f3851dfd7db22d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:29:00.743587 19639 raft_consensus.cc:686] T 00000000000000000000000000000000 P 25c6adebd0e74ef28f3851dfd7db22d8 [term 1 LEADER]: Becoming Leader. State: Replica: 25c6adebd0e74ef28f3851dfd7db22d8, State: Running, Role: LEADER
I20260620 04:29:00.743880 19639 consensus_queue.cc:237] T 00000000000000000000000000000000 P 25c6adebd0e74ef28f3851dfd7db22d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25c6adebd0e74ef28f3851dfd7db22d8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37061 } }
I20260620 04:29:00.744573 19634 sys_catalog.cc:564] T 00000000000000000000000000000000 P 25c6adebd0e74ef28f3851dfd7db22d8 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:29:00.745563 19640 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25c6adebd0e74ef28f3851dfd7db22d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "25c6adebd0e74ef28f3851dfd7db22d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25c6adebd0e74ef28f3851dfd7db22d8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37061 } } }
I20260620 04:29:00.745594 19641 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25c6adebd0e74ef28f3851dfd7db22d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 25c6adebd0e74ef28f3851dfd7db22d8. Latest consensus state: current_term: 1 leader_uuid: "25c6adebd0e74ef28f3851dfd7db22d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25c6adebd0e74ef28f3851dfd7db22d8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37061 } } }
I20260620 04:29:00.745677 19640 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25c6adebd0e74ef28f3851dfd7db22d8 [sys.catalog]: This master's current role is: LEADER
I20260620 04:29:00.745699 19641 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25c6adebd0e74ef28f3851dfd7db22d8 [sys.catalog]: This master's current role is: LEADER
I20260620 04:29:00.746075 19648 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:29:00.748370 19638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:00.735338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33516 (local address 127.0.98.62:37061)
0620 04:29:00.735748 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:00.735767 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:00.735811 (+ 44us) server_negotiation.cc:407] Connection header received
0620 04:29:00.736578 (+ 767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:00.736597 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:00.736917 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:00.737233 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:29:00.737978 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:00.738976 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:00.739808 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:00.740330 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:00.740471 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:00.743069 (+ 2598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:29:00.743073 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:29:00.743089 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:29:00.743120 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:29:00.744951 (+ 1831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:29:00.745435 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:29:00.745441 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:29:00.745448 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:29:00.745592 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:29:00.746028 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:29:00.746032 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:29:00.746034 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:29:00.746576 (+ 542us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:29:00.746785 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0620 04:29:00.747112 (+ 327us) server_negotiation.cc:299] Negotiation successful
0620 04:29:00.747407 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"spinlock_wait_cycles":13696,"thread_start_us":104,"threads_started":1}
I20260620 04:29:00.749356 19648 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:29:00.754590 19648 catalog_manager.cc:1347] Generated new cluster ID: c8bbe5fef228477987bd802ca2ac2dcd
I20260620 04:29:00.754647 19648 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:29:00.763446 19648 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:29:00.764393 19648 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:29:00.777019 19648 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 25c6adebd0e74ef28f3851dfd7db22d8: Generated new TSK 0
I20260620 04:29:00.777693 19648 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929496330398-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929496330398-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 04:29:00.804169 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929496330398-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929496330398-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929496330398-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929496330398-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37061
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929496330398-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:29:00.910004 19660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:29:00.910259 19660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:29:00.910318 19660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:29:00.914273 19660 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:29:00.914320 19660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:29:00.914376 19660 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 04:29:00.919399 19660 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929496330398-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929496330398-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929496330398-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929496330398-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37061
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19660
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929496330398-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:29:00.920356 19660 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:29:00.921290 19660 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:29:00.930301 19665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:29:00.930307 19666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:29:00.930476 19660 server_base.cc:1034] running on GCE node
W20260620 04:29:00.930323 19668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:29:00.931022 19660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:29:00.931617 19660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:29:00.932797 19660 hybrid_clock.cc:648] HybridClock initialized: now 1781929740932777 us; error 30 us; skew 500 ppm
Jun 20 04:29:00 dist-test-slave-nk6d krb5kdc[19553](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929740, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:29:00.935499 19660 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 04:29:00.936400 19660 webserver.cc:460] Webserver started at http://127.0.98.1:38967/ using document root <none> and password file <none>
I20260620 04:29:00.936908 19660 fs_manager.cc:359] Metadata directory not provided
I20260620 04:29:00.936964 19660 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:29:00.937124 19660 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:29:00.939079 19660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929496330398-392-0/minicluster-data/ts-0/data/instance:
uuid: "952660f2f9b0466c9ae09595df58fe4c"
format_stamp: "Formatted at 2026-06-20 04:29:00 on dist-test-slave-nk6d"
server_key: "d9b94cde046ec4d950ed56d38c8010cb"
server_key_iv: "236ee3956dcb89fe840458d076ba9081"
server_key_version: "encryptionkey@0"
I20260620 04:29:00.939615 19660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929496330398-392-0/minicluster-data/ts-0/wal/instance:
uuid: "952660f2f9b0466c9ae09595df58fe4c"
format_stamp: "Formatted at 2026-06-20 04:29:00 on dist-test-slave-nk6d"
server_key: "d9b94cde046ec4d950ed56d38c8010cb"
server_key_iv: "236ee3956dcb89fe840458d076ba9081"
server_key_version: "encryptionkey@0"
I20260620 04:29:00.943183 19660 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 04:29:00.945633 19675 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:29:00.946851 19660 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260620 04:29:00.946944 19660 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929496330398-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929496330398-392-0/minicluster-data/ts-0/wal
uuid: "952660f2f9b0466c9ae09595df58fe4c"
format_stamp: "Formatted at 2026-06-20 04:29:00 on dist-test-slave-nk6d"
server_key: "d9b94cde046ec4d950ed56d38c8010cb"
server_key_iv: "236ee3956dcb89fe840458d076ba9081"
server_key_version: "encryptionkey@0"
I20260620 04:29:00.947041 19660 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929496330398-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929496330398-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929496330398-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:29:00.961414 19660 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:29:00.963917 19660 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:29:00.964100 19660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:29:00.964620 19660 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:29:00.965385 19660 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:29:00.965430 19660 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:29:00.965468 19660 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:29:00.965497 19660 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:29:00.973858 19660 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33641
I20260620 04:29:00.973901 19788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33641 every 8 connection(s)
I20260620 04:29:00.974704 19660 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929496330398-392-0/minicluster-data/ts-0/data/info.pb
I20260620 04:29:00.979547 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 19660
I20260620 04:29:00.979655 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929496330398-392-0/minicluster-data/ts-0/wal/instance
I20260620 04:29:00.979911 392 external_mini_cluster.cc:1428] Setting key f39366f42e44eef37ac77cf9a6aa3ae1
Jun 20 04:29:00 dist-test-slave-nk6d krb5kdc[19553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929740, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:29:00.987329 19638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:00.976409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57843 (local address 127.0.98.62:37061)
0620 04:29:00.976520 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:00.976524 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:00.977308 (+ 784us) server_negotiation.cc:407] Connection header received
0620 04:29:00.978207 (+ 899us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:00.978211 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:00.978269 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:00.978341 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:29:00.979462 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:00.980105 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:00.981393 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:00.981730 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:00.981835 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:00.984261 (+ 2426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:29:00.984264 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:29:00.984266 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:29:00.984287 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:29:00.985507 (+ 1220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:29:00.985962 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:29:00.985965 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:29:00.985967 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:29:00.986037 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:29:00.986325 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:29:00.986328 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:29:00.986330 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:29:00.986493 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:29:00.986575 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 04:29:00.987037 (+ 462us) server_negotiation.cc:299] Negotiation successful
0620 04:29:00.987157 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 04:29:00.988124 19791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:00.976709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37061 (local address 127.0.98.1:57843)
0620 04:29:00.977155 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0620 04:29:00.977196 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 04:29:00.977993 (+ 797us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:29:00.978520 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:29:00.978527 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:29:00.978868 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:29:00.979323 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:29:00.979336 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:29:00.980218 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:29:00.980222 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:29:00.981272 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:29:00.981283 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:29:00.981863 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:29:00.981866 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:29:00.982008 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:00.982558 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:29:00.982578 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:29:00.984111 (+ 1533us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:29:00.985609 (+ 1498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:29:00.985615 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:29:00.985628 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:29:00.985874 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:29:00.986113 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:29:00.986116 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:29:00.986118 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:29:00.986242 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:29:00.986584 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:29:00.986590 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:29:00.986825 (+ 235us) client_negotiation.cc:769] Sending connection context
0620 04:29:00.986983 (+ 158us) client_negotiation.cc:240] Negotiation successful
0620 04:29:00.987285 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":103,"threads_started":1}
I20260620 04:29:00.989246 19789 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37061
I20260620 04:29:00.989475 19789 heartbeater.cc:464] Registering TS with master...
I20260620 04:29:00.989955 19789 heartbeater.cc:511] Master 127.0.98.62:37061 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 04:29:00.991544 19598 ts_manager.cc:194] Registered new tserver with Master: 952660f2f9b0466c9ae09595df58fe4c (127.0.98.1:33641)
I20260620 04:29:00.992705 19598 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:57843
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929496330398-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929496330398-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 04:29:01.003841 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929496330398-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929496330398-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929496330398-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929496330398-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37061
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929496330398-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:29:01.109546 19794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:29:01.109807 19794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:29:01.109864 19794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:29:01.113886 19794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:29:01.113940 19794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:29:01.114018 19794 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 04:29:01.119082 19794 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929496330398-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929496330398-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929496330398-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929496330398-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37061
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19794
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929496330398-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:29:01.120064 19794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:29:01.121006 19794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:29:01.130021 19802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:29:01.130163 19794 server_base.cc:1034] running on GCE node
W20260620 04:29:01.130025 19799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:29:01.130082 19800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:29:01.130590 19794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:29:01.131152 19794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:29:01.132354 19794 hybrid_clock.cc:648] HybridClock initialized: now 1781929741132308 us; error 41 us; skew 500 ppm
Jun 20 04:29:01 dist-test-slave-nk6d krb5kdc[19553](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929741, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:29:01.134933 19794 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 04:29:01.135764 19794 webserver.cc:460] Webserver started at http://127.0.98.2:40373/ using document root <none> and password file <none>
I20260620 04:29:01.136307 19794 fs_manager.cc:359] Metadata directory not provided
I20260620 04:29:01.136391 19794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:29:01.136590 19794 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:29:01.138581 19794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929496330398-392-0/minicluster-data/ts-1/data/instance:
uuid: "10c61d97c22a4152962da97eeced9844"
format_stamp: "Formatted at 2026-06-20 04:29:01 on dist-test-slave-nk6d"
server_key: "cfd9ba0c33648ee572976f97829dc976"
server_key_iv: "819a2cc71954c70fd95805ece44ac1a9"
server_key_version: "encryptionkey@0"
I20260620 04:29:01.139086 19794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929496330398-392-0/minicluster-data/ts-1/wal/instance:
uuid: "10c61d97c22a4152962da97eeced9844"
format_stamp: "Formatted at 2026-06-20 04:29:01 on dist-test-slave-nk6d"
server_key: "cfd9ba0c33648ee572976f97829dc976"
server_key_iv: "819a2cc71954c70fd95805ece44ac1a9"
server_key_version: "encryptionkey@0"
I20260620 04:29:01.142540 19794 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 04:29:01.144848 19809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:29:01.146093 19794 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:29:01.146193 19794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929496330398-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929496330398-392-0/minicluster-data/ts-1/wal
uuid: "10c61d97c22a4152962da97eeced9844"
format_stamp: "Formatted at 2026-06-20 04:29:01 on dist-test-slave-nk6d"
server_key: "cfd9ba0c33648ee572976f97829dc976"
server_key_iv: "819a2cc71954c70fd95805ece44ac1a9"
server_key_version: "encryptionkey@0"
I20260620 04:29:01.146278 19794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929496330398-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929496330398-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929496330398-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:29:01.161514 19794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:29:01.164556 19794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:29:01.164740 19794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:29:01.165277 19794 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:29:01.166078 19794 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:29:01.166126 19794 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:29:01.166165 19794 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:29:01.166179 19794 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:29:01.174500 19794 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41091
I20260620 04:29:01.174515 19922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41091 every 8 connection(s)
I20260620 04:29:01.175370 19794 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929496330398-392-0/minicluster-data/ts-1/data/info.pb
I20260620 04:29:01.179301 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 19794
I20260620 04:29:01.179430 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929496330398-392-0/minicluster-data/ts-1/wal/instance
I20260620 04:29:01.179687 392 external_mini_cluster.cc:1428] Setting key e5f39026194ea4cf58bd45bda8b7e35c
Jun 20 04:29:01 dist-test-slave-nk6d krb5kdc[19553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929741, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:29:01.188663 19638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:01.177119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55329 (local address 127.0.98.62:37061)
0620 04:29:01.177240 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:01.177244 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:01.178077 (+ 833us) server_negotiation.cc:407] Connection header received
0620 04:29:01.178911 (+ 834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:01.178914 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:01.178959 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:01.179031 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:29:01.180317 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:01.181050 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:01.182599 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:01.182830 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:01.182922 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:01.185241 (+ 2319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:29:01.185245 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:29:01.185248 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:29:01.185276 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:29:01.186745 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:29:01.187196 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:29:01.187200 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:29:01.187202 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:29:01.187302 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:29:01.187572 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:29:01.187575 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:29:01.187577 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:29:01.187789 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:29:01.187887 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 04:29:01.188388 (+ 501us) server_negotiation.cc:299] Negotiation successful
0620 04:29:01.188504 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 04:29:01.189361 19925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:01.177459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37061 (local address 127.0.98.2:55329)
0620 04:29:01.177920 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0620 04:29:01.177962 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 04:29:01.178705 (+ 743us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:29:01.179182 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:29:01.179188 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:29:01.179593 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:29:01.180173 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:29:01.180190 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:29:01.181169 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:29:01.181172 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:29:01.182486 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:29:01.182498 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:29:01.182974 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:29:01.182979 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:29:01.183146 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:01.183660 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:29:01.183676 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:29:01.185073 (+ 1397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:29:01.186851 (+ 1778us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:29:01.186856 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:29:01.186867 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:29:01.187092 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:29:01.187381 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:29:01.187383 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:29:01.187384 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:29:01.187496 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:29:01.187895 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:29:01.187900 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:29:01.188154 (+ 254us) client_negotiation.cc:769] Sending connection context
0620 04:29:01.188306 (+ 152us) client_negotiation.cc:240] Negotiation successful
0620 04:29:01.188543 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":107,"threads_started":1}
I20260620 04:29:01.190438 19923 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37061
I20260620 04:29:01.190701 19923 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260620 04:29:01.191232 19923 heartbeater.cc:511] Master 127.0.98.62:37061 requested a full tablet report, sending...
I20260620 04:29:01.192224 19598 ts_manager.cc:194] Registered new tserver with Master: 10c61d97c22a4152962da97eeced9844 (127.0.98.2:41091)
I20260620 04:29:01.192766 19598 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:55329
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929496330398-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929496330398-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 04:29:01.204706 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929496330398-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929496330398-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929496330398-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929496330398-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37061
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929496330398-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:29:01.309791 19928 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:29:01.310040 19928 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:29:01.310097 19928 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:29:01.314177 19928 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:29:01.314237 19928 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:29:01.314316 19928 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 04:29:01.319340 19928 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929496330398-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929496330398-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929496330398-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929496330398-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37061
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19928
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929496330398-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:29:01.320313 19928 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:29:01.321256 19928 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:29:01.330295 19936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:29:01.330308 19933 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:29:01.330303 19934 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:29:01.330364 19928 server_base.cc:1034] running on GCE node
I20260620 04:29:01.330912 19928 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:29:01.331496 19928 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:29:01.332654 19928 hybrid_clock.cc:648] HybridClock initialized: now 1781929741332632 us; error 35 us; skew 500 ppm
Jun 20 04:29:01 dist-test-slave-nk6d krb5kdc[19553](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929741, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:29:01.335253 19928 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 04:29:01.336084 19928 webserver.cc:460] Webserver started at http://127.0.98.3:40421/ using document root <none> and password file <none>
I20260620 04:29:01.336582 19928 fs_manager.cc:359] Metadata directory not provided
I20260620 04:29:01.336638 19928 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:29:01.336798 19928 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:29:01.338726 19928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929496330398-392-0/minicluster-data/ts-2/data/instance:
uuid: "f2513956ba18490e91cc1e2735d87f79"
format_stamp: "Formatted at 2026-06-20 04:29:01 on dist-test-slave-nk6d"
server_key: "22f0508457167e787f25f39e2463909e"
server_key_iv: "5fb2c26fb6e7dff40bbf9a451eeb2016"
server_key_version: "encryptionkey@0"
I20260620 04:29:01.339210 19928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929496330398-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f2513956ba18490e91cc1e2735d87f79"
format_stamp: "Formatted at 2026-06-20 04:29:01 on dist-test-slave-nk6d"
server_key: "22f0508457167e787f25f39e2463909e"
server_key_iv: "5fb2c26fb6e7dff40bbf9a451eeb2016"
server_key_version: "encryptionkey@0"
I20260620 04:29:01.342651 19928 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.006s
I20260620 04:29:01.344918 19943 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:29:01.346123 19928 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 04:29:01.346215 19928 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929496330398-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929496330398-392-0/minicluster-data/ts-2/wal
uuid: "f2513956ba18490e91cc1e2735d87f79"
format_stamp: "Formatted at 2026-06-20 04:29:01 on dist-test-slave-nk6d"
server_key: "22f0508457167e787f25f39e2463909e"
server_key_iv: "5fb2c26fb6e7dff40bbf9a451eeb2016"
server_key_version: "encryptionkey@0"
I20260620 04:29:01.346333 19928 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929496330398-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929496330398-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929496330398-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:29:01.356966 19928 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:29:01.359730 19928 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:29:01.359885 19928 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:29:01.360428 19928 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:29:01.361171 19928 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:29:01.361214 19928 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:29:01.361253 19928 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:29:01.361271 19928 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:29:01.369596 19928 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42717
I20260620 04:29:01.369621 20056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42717 every 8 connection(s)
I20260620 04:29:01.370437 19928 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929496330398-392-0/minicluster-data/ts-2/data/info.pb
Jun 20 04:29:01 dist-test-slave-nk6d krb5kdc[19553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929741, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:29:01.380069 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 19928
I20260620 04:29:01.380199 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929496330398-392-0/minicluster-data/ts-2/wal/instance
I20260620 04:29:01.380546 392 external_mini_cluster.cc:1428] Setting key 08da7aae7d3c5452550fd9b40e49bab4
I20260620 04:29:01.382817 19638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:01.372180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44399 (local address 127.0.98.62:37061)
0620 04:29:01.372347 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:01.372352 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:01.373193 (+ 841us) server_negotiation.cc:407] Connection header received
0620 04:29:01.374027 (+ 834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:01.374030 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:01.374076 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:01.374149 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:29:01.375257 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:01.375769 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:01.376900 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:01.377111 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:01.377188 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:01.379362 (+ 2174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:29:01.379365 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:29:01.379367 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:29:01.379390 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:29:01.380894 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:29:01.381379 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:29:01.381383 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:29:01.381385 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:29:01.381468 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:29:01.381811 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:29:01.381815 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:29:01.381816 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:29:01.381989 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:29:01.382061 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 04:29:01.382615 (+ 554us) server_negotiation.cc:299] Negotiation successful
0620 04:29:01.382705 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 04:29:01.383636 20059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:01.372566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37061 (local address 127.0.98.3:44399)
0620 04:29:01.373031 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0620 04:29:01.373073 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 04:29:01.373836 (+ 763us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:29:01.374314 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:29:01.374320 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:29:01.374655 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:29:01.375127 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:29:01.375140 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:29:01.375872 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:29:01.375875 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:29:01.376794 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:29:01.376802 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:29:01.377223 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:29:01.377225 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:29:01.377363 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:01.377869 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:29:01.377885 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:29:01.379214 (+ 1329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:29:01.380991 (+ 1777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:29:01.380997 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:29:01.381010 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:29:01.381289 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:29:01.381550 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:29:01.381552 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:29:01.381554 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:29:01.381726 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:29:01.382081 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:29:01.382087 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:29:01.382404 (+ 317us) client_negotiation.cc:769] Sending connection context
0620 04:29:01.382575 (+ 171us) client_negotiation.cc:240] Negotiation successful
0620 04:29:01.382825 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":111,"threads_started":1}
I20260620 04:29:01.384620 20057 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37061
I20260620 04:29:01.384811 20057 heartbeater.cc:464] Registering TS with master...
I20260620 04:29:01.385234 20057 heartbeater.cc:511] Master 127.0.98.62:37061 requested a full tablet report, sending...
I20260620 04:29:01.386101 19598 ts_manager.cc:194] Registered new tserver with Master: f2513956ba18490e91cc1e2735d87f79 (127.0.98.3:42717)
I20260620 04:29:01.386620 19598 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:44399
I20260620 04:29:01.394045 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 04:29:01.402257 19638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:01.395775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33522 (local address 127.0.98.62:37061)
0620 04:29:01.395889 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:01.395893 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:01.396044 (+ 151us) server_negotiation.cc:407] Connection header received
0620 04:29:01.396135 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:01.396137 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:01.396174 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:01.396236 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:29:01.396838 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:01.397330 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:01.398070 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:01.398269 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:01.398355 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:01.399545 (+ 1190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:29:01.399548 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:29:01.399550 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:29:01.399570 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:29:01.400792 (+ 1222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:29:01.401152 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:29:01.401154 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:29:01.401155 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:29:01.401226 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:29:01.401422 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:29:01.401424 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:29:01.401425 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:29:01.401618 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:29:01.401713 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 04:29:01.402005 (+ 292us) server_negotiation.cc:299] Negotiation successful
0620 04:29:01.402102 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 04:29:01.404908 19598 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33522:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 04:29:01.407819 19598 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 04:29:01.421058 20068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:01.415649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41091 (local address 127.0.0.1:56708)
0620 04:29:01.416089 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0620 04:29:01.416102 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:29:01.416187 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:29:01.417180 (+ 993us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:29:01.417187 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:29:01.417210 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:29:01.417470 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:29:01.417475 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:29:01.418384 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:29:01.418388 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:29:01.419409 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:29:01.419419 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:29:01.420527 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:29:01.420531 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:29:01.420636 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:01.420674 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 04:29:01.420784 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 04:29:01.420907 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377,"mutex_wait_us":112,"spinlock_wait_cycles":768,"thread_start_us":107,"threads_started":1}
I20260620 04:29:01.421066 20072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:01.416351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56708 (local address 127.0.98.2:41091)
0620 04:29:01.416681 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:01.416692 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:01.416716 (+ 24us) server_negotiation.cc:407] Connection header received
0620 04:29:01.416814 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:01.416820 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:01.417025 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:01.417418 (+ 393us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:29:01.417566 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:01.418269 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:01.419536 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:01.420409 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:01.420512 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:01.420653 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 04:29:01.420821 (+ 168us) server_negotiation.cc:299] Negotiation successful
0620 04:29:01.420916 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":112,"threads_started":1}
I20260620 04:29:01.421417 20069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:01.415554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42717 (local address 127.0.0.1:42080)
0620 04:29:01.416016 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0620 04:29:01.416060 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 04:29:01.416179 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:29:01.417229 (+ 1050us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:29:01.417232 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:29:01.417259 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:29:01.417373 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:29:01.417380 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:29:01.419001 (+ 1621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:29:01.419004 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:29:01.420127 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:29:01.420136 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:29:01.421076 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:29:01.421079 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:29:01.421160 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:01.421184 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 04:29:01.421231 (+ 47us) client_negotiation.cc:240] Negotiation successful
0620 04:29:01.421303 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"mutex_wait_us":56,"spinlock_wait_cycles":768,"thread_start_us":77,"threads_started":1}
I20260620 04:29:01.421852 20071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:01.416256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33641 (local address 127.0.0.1:56866)
0620 04:29:01.416475 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0620 04:29:01.416488 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:29:01.416565 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:29:01.417301 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:29:01.417303 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:29:01.417315 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:29:01.417394 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:29:01.417400 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:29:01.419026 (+ 1626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:29:01.419030 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:29:01.420114 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:29:01.420125 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:29:01.421503 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:29:01.421507 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:29:01.421591 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:01.421614 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 04:29:01.421658 (+ 44us) client_negotiation.cc:240] Negotiation successful
0620 04:29:01.421716 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"mutex_wait_us":26,"thread_start_us":88,"threads_started":1}
I20260620 04:29:01.422008 20070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:01.415676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42080 (local address 127.0.98.3:42717)
0620 04:29:01.416775 (+ 1099us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:01.416786 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:01.416809 (+ 23us) server_negotiation.cc:407] Connection header received
0620 04:29:01.416880 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:01.416885 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:01.417075 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:01.417521 (+ 446us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:29:01.417680 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:01.418861 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:01.420252 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:01.420956 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:01.421651 (+ 695us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:01.421758 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 04:29:01.421799 (+ 41us) server_negotiation.cc:299] Negotiation successful
0620 04:29:01.421867 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1004,"thread_start_us":142,"threads_started":1}
I20260620 04:29:01.422490 20073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:01.416627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56866 (local address 127.0.98.1:33641)
0620 04:29:01.416930 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:01.416943 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:01.416959 (+ 16us) server_negotiation.cc:407] Connection header received
0620 04:29:01.417020 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:01.417024 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:01.417164 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:01.417584 (+ 420us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:29:01.417720 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:01.418861 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:01.420246 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:01.421037 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:01.422072 (+ 1035us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:01.422214 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 04:29:01.422263 (+ 49us) server_negotiation.cc:299] Negotiation successful
0620 04:29:01.422352 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":109,"threads_started":1}
I20260620 04:29:01.424021 19857 tablet_service.cc:1467] Processing CreateTablet for tablet 60fd15a522bf4e6eaf66d5faf9bd2a33 (DEFAULT_TABLE table=test-table [id=24a42a1d7c954b1a99e7acd973c536a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:29:01.424444 19991 tablet_service.cc:1467] Processing CreateTablet for tablet 60fd15a522bf4e6eaf66d5faf9bd2a33 (DEFAULT_TABLE table=test-table [id=24a42a1d7c954b1a99e7acd973c536a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:29:01.424934 19857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60fd15a522bf4e6eaf66d5faf9bd2a33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:29:01.425295 19991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60fd15a522bf4e6eaf66d5faf9bd2a33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:29:01.430967 20074 tablet_bootstrap.cc:492] T 60fd15a522bf4e6eaf66d5faf9bd2a33 P f2513956ba18490e91cc1e2735d87f79: Bootstrap starting.
I20260620 04:29:01.432191 20075 tablet_bootstrap.cc:492] T 60fd15a522bf4e6eaf66d5faf9bd2a33 P 10c61d97c22a4152962da97eeced9844: Bootstrap starting.
I20260620 04:29:01.432065 19723 tablet_service.cc:1467] Processing CreateTablet for tablet 60fd15a522bf4e6eaf66d5faf9bd2a33 (DEFAULT_TABLE table=test-table [id=24a42a1d7c954b1a99e7acd973c536a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:29:01.432993 20074 tablet_bootstrap.cc:654] T 60fd15a522bf4e6eaf66d5faf9bd2a33 P f2513956ba18490e91cc1e2735d87f79: Neither blocks nor log segments found. Creating new log.
I20260620 04:29:01.433145 19723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60fd15a522bf4e6eaf66d5faf9bd2a33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:29:01.433917 20074 log.cc:824] T 60fd15a522bf4e6eaf66d5faf9bd2a33 P f2513956ba18490e91cc1e2735d87f79: Log is configured to *not* fsync() on all Append() calls
I20260620 04:29:01.434249 20075 tablet_bootstrap.cc:654] T 60fd15a522bf4e6eaf66d5faf9bd2a33 P 10c61d97c22a4152962da97eeced9844: Neither blocks nor log segments found. Creating new log.
I20260620 04:29:01.435341 20075 log.cc:824] T 60fd15a522bf4e6eaf66d5faf9bd2a33 P 10c61d97c22a4152962da97eeced9844: Log is configured to *not* fsync() on all Append() calls
I20260620 04:29:01.435527 20074 tablet_bootstrap.cc:492] T 60fd15a522bf4e6eaf66d5faf9bd2a33 P f2513956ba18490e91cc1e2735d87f79: No bootstrap required, opened a new log
I20260620 04:29:01.435667 20074 ts_tablet_manager.cc:1397] T 60fd15a522bf4e6eaf66d5faf9bd2a33 P f2513956ba18490e91cc1e2735d87f79: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 04:29:01.437322 20075 tablet_bootstrap.cc:492] T 60fd15a522bf4e6eaf66d5faf9bd2a33 P 10c61d97c22a4152962da97eeced9844: No bootstrap required, opened a new log
I20260620 04:29:01.437484 20075 ts_tablet_manager.cc:1397] T 60fd15a522bf4e6eaf66d5faf9bd2a33 P 10c61d97c22a4152962da97eeced9844: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 04:29:01.438519 20074 raft_consensus.cc:348] T 60fd15a522bf4e6eaf66d5faf9bd2a33 P f2513956ba18490e91cc1e2735d87f79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2513956ba18490e91cc1e2735d87f79" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42717 } } peers { permanent_uuid: "10c61d97c22a4152962da97eeced9844" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41091 } } peers { permanent_uuid: "952660f2f9b0466c9ae09595df58fe4c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33641 } }
I20260620 04:29:01.438719 20074 raft_consensus.cc:374] T 60fd15a522bf4e6eaf66d5faf9bd2a33 P f2513956ba18490e91cc1e2735d87f79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:29:01.438755 20074 raft_consensus.cc:729] T 60fd15a522bf4e6eaf66d5faf9bd2a33 P f2513956ba18490e91cc1e2735d87f79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2513956ba18490e91cc1e2735d87f79, State: Initialized, Role: FOLLOWER
I20260620 04:29:01.439141 20074 consensus_queue.cc:260] T 60fd15a522bf4e6eaf66d5faf9bd2a33 P f2513956ba18490e91cc1e2735d87f79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2513956ba18490e91cc1e2735d87f79" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42717 } } peers { permanent_uuid: "10c61d97c22a4152962da97eeced9844" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41091 } } peers { permanent_uuid: "952660f2f9b0466c9ae09595df58fe4c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33641 } }
I20260620 04:29:01.440325 20057 heartbeater.cc:503] Master 127.0.98.62:37061 was elected leader, sending a full tablet report...
I20260620 04:29:01.440595 20078 tablet_bootstrap.cc:492] T 60fd15a522bf4e6eaf66d5faf9bd2a33 P 952660f2f9b0466c9ae09595df58fe4c: Bootstrap starting.
I20260620 04:29:01.441092 20075 raft_consensus.cc:348] T 60fd15a522bf4e6eaf66d5faf9bd2a33 P 10c61d97c22a4152962da97eeced9844 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2513956ba18490e91cc1e2735d87f79" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42717 } } peers { permanent_uuid: "10c61d97c22a4152962da97eeced9844" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41091 } } peers { permanent_uuid: "952660f2f9b0466c9ae09595df58fe4c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33641 } }
I20260620 04:29:01.441285 20074 ts_tablet_manager.cc:1428] T 60fd15a522bf4e6eaf66d5faf9bd2a33 P f2513956ba18490e91cc1e2735d87f79: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260620 04:29:01.441358 20075 raft_consensus.cc:374] T 60fd15a522bf4e6eaf66d5faf9bd2a33 P 10c61d97c22a4152962da97eeced9844 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:29:01.441409 20075 raft_consensus.cc:729] T 60fd15a522bf4e6eaf66d5faf9bd2a33 P 10c61d97c22a4152962da97eeced9844 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10c61d97c22a4152962da97eeced9844, State: Initialized, Role: FOLLOWER
I20260620 04:29:01.441869 20075 consensus_queue.cc:260] T 60fd15a522bf4e6eaf66d5faf9bd2a33 P 10c61d97c22a4152962da97eeced9844 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2513956ba18490e91cc1e2735d87f79" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42717 } } peers { permanent_uuid: "10c61d97c22a4152962da97eeced9844" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41091 } } peers { permanent_uuid: "952660f2f9b0466c9ae09595df58fe4c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33641 } }
I20260620 04:29:01.442433 19923 heartbeater.cc:503] Master 127.0.98.62:37061 was elected leader, sending a full tablet report...
I20260620 04:29:01.442996 20078 tablet_bootstrap.cc:654] T 60fd15a522bf4e6eaf66d5faf9bd2a33 P 952660f2f9b0466c9ae09595df58fe4c: Neither blocks nor log segments found. Creating new log.
I20260620 04:29:01.443207 20080 raft_consensus.cc:482] T 60fd15a522bf4e6eaf66d5faf9bd2a33 P 10c61d97c22a4152962da97eeced9844 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:29:01.443336 20080 raft_consensus.cc:504] T 60fd15a522bf4e6eaf66d5faf9bd2a33 P 10c61d97c22a4152962da97eeced9844 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2513956ba18490e91cc1e2735d87f79" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42717 } } peers { permanent_uuid: "10c61d97c22a4152962da97eeced9844" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41091 } } peers { permanent_uuid: "952660f2f9b0466c9ae09595df58fe4c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33641 } }
I20260620 04:29:01.444191 20078 log.cc:824] T 60fd15a522bf4e6eaf66d5faf9bd2a33 P 952660f2f9b0466c9ae09595df58fe4c: Log is configured to *not* fsync() on all Append() calls
I20260620 04:29:01.444777 20075 ts_tablet_manager.cc:1428] T 60fd15a522bf4e6eaf66d5faf9bd2a33 P 10c61d97c22a4152962da97eeced9844: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260620 04:29:01.445962 20080 leader_election.cc:288] T 60fd15a522bf4e6eaf66d5faf9bd2a33 P 10c61d97c22a4152962da97eeced9844 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f2513956ba18490e91cc1e2735d87f79 (127.0.98.3:42717), 952660f2f9b0466c9ae09595df58fe4c (127.0.98.1:33641)
I20260620 04:29:01.446318 20078 tablet_bootstrap.cc:492] T 60fd15a522bf4e6eaf66d5faf9bd2a33 P 952660f2f9b0466c9ae09595df58fe4c: No bootstrap required, opened a new log
I20260620 04:29:01.446491 20078 ts_tablet_manager.cc:1397] T 60fd15a522bf4e6eaf66d5faf9bd2a33 P 952660f2f9b0466c9ae09595df58fe4c: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 04:29:01.450065 20078 raft_consensus.cc:348] T 60fd15a522bf4e6eaf66d5faf9bd2a33 P 952660f2f9b0466c9ae09595df58fe4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2513956ba18490e91cc1e2735d87f79" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42717 } } peers { permanent_uuid: "10c61d97c22a4152962da97eeced9844" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41091 } } peers { permanent_uuid: "952660f2f9b0466c9ae09595df58fe4c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33641 } }
I20260620 04:29:01.450325 20083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:01.446396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33641 (local address 127.0.98.2:40285)
0620 04:29:01.446689 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0620 04:29:01.446702 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:29:01.446911 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:29:01.447171 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:29:01.447174 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:29:01.447192 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:29:01.447307 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:29:01.447313 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:29:01.448147 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:29:01.448150 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:29:01.449060 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:29:01.449069 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:29:01.449912 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:29:01.449916 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:29:01.450012 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:01.450038 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 04:29:01.450118 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 04:29:01.450185 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":79,"threads_started":1}
I20260620 04:29:01.450321 20073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:01.446769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40285 (local address 127.0.98.1:33641)
0620 04:29:01.446870 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:01.446874 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:01.446888 (+ 14us) server_negotiation.cc:407] Connection header received
0620 04:29:01.447019 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:01.447022 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:01.447072 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:01.447137 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:29:01.447408 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:01.448056 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:01.449159 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:01.449799 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:01.449911 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:01.449988 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 04:29:01.450133 (+ 145us) server_negotiation.cc:299] Negotiation successful
0620 04:29:01.450201 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 04:29:01.450326 20078 raft_consensus.cc:374] T 60fd15a522bf4e6eaf66d5faf9bd2a33 P 952660f2f9b0466c9ae09595df58fe4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:29:01.450409 20078 raft_consensus.cc:729] T 60fd15a522bf4e6eaf66d5faf9bd2a33 P 952660f2f9b0466c9ae09595df58fe4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 952660f2f9b0466c9ae09595df58fe4c, State: Initialized, Role: FOLLOWER
I20260620 04:29:01.450544 19925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:01.446165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42717 (local address 127.0.98.2:48513)
0620 04:29:01.446337 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 04:29:01.446351 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:29:01.446422 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:29:01.446840 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:29:01.446843 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:29:01.446861 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:29:01.446944 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:29:01.446949 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:29:01.447833 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:29:01.447836 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:29:01.449165 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:29:01.449175 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:29:01.450153 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:29:01.450157 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:29:01.450272 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:01.450294 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 04:29:01.450361 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 04:29:01.450428 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100}
I20260620 04:29:01.450855 20078 consensus_queue.cc:260] T 60fd15a522bf4e6eaf66d5faf9bd2a33 P 952660f2f9b0466c9ae09595df58fe4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2513956ba18490e91cc1e2735d87f79" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42717 } } peers { permanent_uuid: "10c61d97c22a4152962da97eeced9844" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41091 } } peers { permanent_uuid: "952660f2f9b0466c9ae09595df58fe4c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33641 } }
I20260620 04:29:01.450964 20070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:01.446226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48513 (local address 127.0.98.3:42717)
0620 04:29:01.446565 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:01.446569 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:01.446581 (+ 12us) server_negotiation.cc:407] Connection header received
0620 04:29:01.446621 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:01.446624 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:01.446665 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:01.446725 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:29:01.447053 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:01.447735 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:01.449290 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:01.449942 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:01.450025 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:01.450076 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 04:29:01.450773 (+ 697us) server_negotiation.cc:299] Negotiation successful
0620 04:29:01.450862 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271}
I20260620 04:29:01.451192 19743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60fd15a522bf4e6eaf66d5faf9bd2a33" candidate_uuid: "10c61d97c22a4152962da97eeced9844" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "952660f2f9b0466c9ae09595df58fe4c" is_pre_election: true
I20260620 04:29:01.451393 19743 raft_consensus.cc:2454] T 60fd15a522bf4e6eaf66d5faf9bd2a33 P 952660f2f9b0466c9ae09595df58fe4c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10c61d97c22a4152962da97eeced9844 in term 0.
I20260620 04:29:01.451433 20011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60fd15a522bf4e6eaf66d5faf9bd2a33" candidate_uuid: "10c61d97c22a4152962da97eeced9844" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2513956ba18490e91cc1e2735d87f79" is_pre_election: true
I20260620 04:29:01.451676 20011 raft_consensus.cc:2454] T 60fd15a522bf4e6eaf66d5faf9bd2a33 P f2513956ba18490e91cc1e2735d87f79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10c61d97c22a4152962da97eeced9844 in term 0.
I20260620 04:29:01.451743 20078 ts_tablet_manager.cc:1428] T 60fd15a522bf4e6eaf66d5faf9bd2a33 P 952660f2f9b0466c9ae09595df58fe4c: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 04:29:01.451809 19810 leader_election.cc:302] T 60fd15a522bf4e6eaf66d5faf9bd2a33 P 10c61d97c22a4152962da97eeced9844 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10c61d97c22a4152962da97eeced9844, 952660f2f9b0466c9ae09595df58fe4c; no voters:
I20260620 04:29:01.452066 19789 heartbeater.cc:503] Master 127.0.98.62:37061 was elected leader, sending a full tablet report...
I20260620 04:29:01.452280 20080 raft_consensus.cc:2781] T 60fd15a522bf4e6eaf66d5faf9bd2a33 P 10c61d97c22a4152962da97eeced9844 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 04:29:01.452345 20080 raft_consensus.cc:482] T 60fd15a522bf4e6eaf66d5faf9bd2a33 P 10c61d97c22a4152962da97eeced9844 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 04:29:01.452394 20080 raft_consensus.cc:3037] T 60fd15a522bf4e6eaf66d5faf9bd2a33 P 10c61d97c22a4152962da97eeced9844 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:29:01.453501 20080 raft_consensus.cc:504] T 60fd15a522bf4e6eaf66d5faf9bd2a33 P 10c61d97c22a4152962da97eeced9844 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2513956ba18490e91cc1e2735d87f79" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42717 } } peers { permanent_uuid: "10c61d97c22a4152962da97eeced9844" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41091 } } peers { permanent_uuid: "952660f2f9b0466c9ae09595df58fe4c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33641 } }
I20260620 04:29:01.453810 20080 leader_election.cc:288] T 60fd15a522bf4e6eaf66d5faf9bd2a33 P 10c61d97c22a4152962da97eeced9844 [CANDIDATE]: Term 1 election: Requested vote from peers f2513956ba18490e91cc1e2735d87f79 (127.0.98.3:42717), 952660f2f9b0466c9ae09595df58fe4c (127.0.98.1:33641)
I20260620 04:29:01.454137 19743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60fd15a522bf4e6eaf66d5faf9bd2a33" candidate_uuid: "10c61d97c22a4152962da97eeced9844" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "952660f2f9b0466c9ae09595df58fe4c"
I20260620 04:29:01.454247 19743 raft_consensus.cc:3037] T 60fd15a522bf4e6eaf66d5faf9bd2a33 P 952660f2f9b0466c9ae09595df58fe4c [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:29:01.455291 19743 raft_consensus.cc:2454] T 60fd15a522bf4e6eaf66d5faf9bd2a33 P 952660f2f9b0466c9ae09595df58fe4c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10c61d97c22a4152962da97eeced9844 in term 1.
I20260620 04:29:01.455494 20011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60fd15a522bf4e6eaf66d5faf9bd2a33" candidate_uuid: "10c61d97c22a4152962da97eeced9844" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2513956ba18490e91cc1e2735d87f79"
I20260620 04:29:01.455520 19810 leader_election.cc:302] T 60fd15a522bf4e6eaf66d5faf9bd2a33 P 10c61d97c22a4152962da97eeced9844 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10c61d97c22a4152962da97eeced9844, 952660f2f9b0466c9ae09595df58fe4c; no voters:
I20260620 04:29:01.455583 20011 raft_consensus.cc:3037] T 60fd15a522bf4e6eaf66d5faf9bd2a33 P f2513956ba18490e91cc1e2735d87f79 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:29:01.455653 20080 raft_consensus.cc:2781] T 60fd15a522bf4e6eaf66d5faf9bd2a33 P 10c61d97c22a4152962da97eeced9844 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:29:01.455741 20080 raft_consensus.cc:686] T 60fd15a522bf4e6eaf66d5faf9bd2a33 P 10c61d97c22a4152962da97eeced9844 [term 1 LEADER]: Becoming Leader. State: Replica: 10c61d97c22a4152962da97eeced9844, State: Running, Role: LEADER
I20260620 04:29:01.456059 20080 consensus_queue.cc:237] T 60fd15a522bf4e6eaf66d5faf9bd2a33 P 10c61d97c22a4152962da97eeced9844 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2513956ba18490e91cc1e2735d87f79" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42717 } } peers { permanent_uuid: "10c61d97c22a4152962da97eeced9844" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41091 } } peers { permanent_uuid: "952660f2f9b0466c9ae09595df58fe4c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33641 } }
I20260620 04:29:01.456635 20011 raft_consensus.cc:2454] T 60fd15a522bf4e6eaf66d5faf9bd2a33 P f2513956ba18490e91cc1e2735d87f79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10c61d97c22a4152962da97eeced9844 in term 1.
I20260620 04:29:01.458703 19598 catalog_manager.cc:5495] T 60fd15a522bf4e6eaf66d5faf9bd2a33 P 10c61d97c22a4152962da97eeced9844 reported cstate change: term changed from 0 to 1, leader changed from <none> to 10c61d97c22a4152962da97eeced9844 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "10c61d97c22a4152962da97eeced9844" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2513956ba18490e91cc1e2735d87f79" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10c61d97c22a4152962da97eeced9844" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "952660f2f9b0466c9ae09595df58fe4c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33641 } health_report { overall_health: UNKNOWN } } }
I20260620 04:29:01.469098 20072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:01.465739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56714 (local address 127.0.98.2:41091)
0620 04:29:01.465856 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:01.465860 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:01.465873 (+ 13us) server_negotiation.cc:407] Connection header received
0620 04:29:01.465988 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:01.465990 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:01.466039 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:01.466120 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:29:01.466347 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:01.466843 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:01.467668 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:01.467898 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:01.468003 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:01.468267 (+ 264us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:29:01.468732 (+ 465us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:29:01.468847 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 04:29:01.468901 (+ 54us) server_negotiation.cc:299] Negotiation successful
0620 04:29:01.468953 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 04:29:01.474355 19743 raft_consensus.cc:1261] T 60fd15a522bf4e6eaf66d5faf9bd2a33 P 952660f2f9b0466c9ae09595df58fe4c [term 1 FOLLOWER]: Refusing update from remote peer 10c61d97c22a4152962da97eeced9844: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:29:01.474357 20011 raft_consensus.cc:1261] T 60fd15a522bf4e6eaf66d5faf9bd2a33 P f2513956ba18490e91cc1e2735d87f79 [term 1 FOLLOWER]: Refusing update from remote peer 10c61d97c22a4152962da97eeced9844: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:29:01.475025 20080 consensus_queue.cc:1035] T 60fd15a522bf4e6eaf66d5faf9bd2a33 P 10c61d97c22a4152962da97eeced9844 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2513956ba18490e91cc1e2735d87f79" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:29:01.475206 20081 consensus_queue.cc:1035] T 60fd15a522bf4e6eaf66d5faf9bd2a33 P 10c61d97c22a4152962da97eeced9844 [LEADER]: Connected to new peer: Peer: permanent_uuid: "952660f2f9b0466c9ae09595df58fe4c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 04:29:01.475726 19790 tablet.cc:2369] T 60fd15a522bf4e6eaf66d5faf9bd2a33 P 952660f2f9b0466c9ae09595df58fe4c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 04:29:01.490463 19598 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33522:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:29:01.490648 19598 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33522:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:29:01.492880 19598 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 25c6adebd0e74ef28f3851dfd7db22d8: Sending DeleteTablet for 3 replicas of tablet 60fd15a522bf4e6eaf66d5faf9bd2a33
I20260620 04:29:01.493718 19857 tablet_service.cc:1514] Processing DeleteTablet for tablet 60fd15a522bf4e6eaf66d5faf9bd2a33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:29:01 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56708
I20260620 04:29:01.493713 19991 tablet_service.cc:1514] Processing DeleteTablet for tablet 60fd15a522bf4e6eaf66d5faf9bd2a33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:29:01 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42080
I20260620 04:29:01.493714 19723 tablet_service.cc:1514] Processing DeleteTablet for tablet 60fd15a522bf4e6eaf66d5faf9bd2a33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:29:01 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56866
I20260620 04:29:01.494109 20097 tablet_replica.cc:330] T 60fd15a522bf4e6eaf66d5faf9bd2a33 P 952660f2f9b0466c9ae09595df58fe4c: stopping tablet replica
I20260620 04:29:01.494354 20099 tablet_replica.cc:330] T 60fd15a522bf4e6eaf66d5faf9bd2a33 P f2513956ba18490e91cc1e2735d87f79: stopping tablet replica
I20260620 04:29:01.494443 20097 raft_consensus.cc:2229] T 60fd15a522bf4e6eaf66d5faf9bd2a33 P 952660f2f9b0466c9ae09595df58fe4c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:29:01.494632 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 19660
I20260620 04:29:01.494627 20099 raft_consensus.cc:2229] T 60fd15a522bf4e6eaf66d5faf9bd2a33 P f2513956ba18490e91cc1e2735d87f79 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:29:01.495338 20098 tablet_replica.cc:330] T 60fd15a522bf4e6eaf66d5faf9bd2a33 P 10c61d97c22a4152962da97eeced9844: stopping tablet replica
I20260620 04:29:01.495589 20098 raft_consensus.cc:2229] T 60fd15a522bf4e6eaf66d5faf9bd2a33 P 10c61d97c22a4152962da97eeced9844 [term 1 LEADER]: Raft consensus shutting down.
I20260620 04:29:01.495937 20098 raft_consensus.cc:2258] T 60fd15a522bf4e6eaf66d5faf9bd2a33 P 10c61d97c22a4152962da97eeced9844 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:29:01.496332 20099 raft_consensus.cc:2258] T 60fd15a522bf4e6eaf66d5faf9bd2a33 P f2513956ba18490e91cc1e2735d87f79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:29:01.497867 20099 ts_tablet_manager.cc:1905] T 60fd15a522bf4e6eaf66d5faf9bd2a33 P f2513956ba18490e91cc1e2735d87f79: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:29:01.500599 20099 ts_tablet_manager.cc:1918] T 60fd15a522bf4e6eaf66d5faf9bd2a33 P f2513956ba18490e91cc1e2735d87f79: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:29:01.500692 20099 log.cc:1196] T 60fd15a522bf4e6eaf66d5faf9bd2a33 P f2513956ba18490e91cc1e2735d87f79: Deleting WAL directory at /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929496330398-392-0/minicluster-data/ts-2/wal/wals/60fd15a522bf4e6eaf66d5faf9bd2a33
W20260620 04:29:01.500885 19582 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33641 (error 108)
I20260620 04:29:01.500994 20099 ts_tablet_manager.cc:1939] T 60fd15a522bf4e6eaf66d5faf9bd2a33 P f2513956ba18490e91cc1e2735d87f79: Deleting consensus metadata
I20260620 04:29:01.501453 20098 ts_tablet_manager.cc:1905] T 60fd15a522bf4e6eaf66d5faf9bd2a33 P 10c61d97c22a4152962da97eeced9844: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:29:01.502135 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 19794
I20260620 04:29:01.502190 20071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:01.501908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33641 (local address 127.0.0.1:56878)
0620 04:29:01.502034 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 04:29:01.502106 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260620 04:29:01.502743 19582 catalog_manager.cc:4584] TS 952660f2f9b0466c9ae09595df58fe4c (127.0.98.1:33641): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 60fd15a522bf4e6eaf66d5faf9bd2a33: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33641: connect: Connection refused (error 111)
I20260620 04:29:01.503037 19582 catalog_manager.cc:4841] TS f2513956ba18490e91cc1e2735d87f79 (127.0.98.3:42717): tablet 60fd15a522bf4e6eaf66d5faf9bd2a33 (table test-table [id=24a42a1d7c954b1a99e7acd973c536a0]) successfully deleted
I20260620 04:29:01.507151 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 19928
I20260620 04:29:01.508569 20071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:01.507176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41091 (local address 127.0.0.1:56726)
0620 04:29:01.508446 (+ 1270us) negotiation.cc:107] Waiting for socket to connect
0620 04:29:01.508497 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:41091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1192}
I20260620 04:29:01.511790 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 19566
2026-06-20T04:29:01Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2008 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-20T04:29:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:29:01Z Disabled control of system clock
I20260620 04:29:01.540876 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929496330398-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929496330398-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929496330398-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42191
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:39543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42191
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 04:29:01.651345 20104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:29:01.651613 20104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:29:01.651649 20104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:29:01.655663 20104 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:29:01.655722 20104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:29:01.655738 20104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:29:01.655755 20104 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:29:01.655782 20104 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:29:01.660532 20104 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929496330398-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929496330398-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42191
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42191
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20104
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929496330398-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:29:01.661659 20104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:29:01.663014 20104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:29:01.671244 20109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:29:01.671250 20110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:29:01.671645 20112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:29:01.672138 20104 server_base.cc:1034] running on GCE node
I20260620 04:29:01.672852 20104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:29:01.674293 20104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:29:01.675447 20104 hybrid_clock.cc:648] HybridClock initialized: now 1781929741675424 us; error 39 us; skew 500 ppm
I20260620 04:29:01.677203 20104 webserver.cc:460] Webserver started at http://127.0.98.62:38417/ using document root <none> and password file <none>
I20260620 04:29:01.677754 20104 fs_manager.cc:359] Metadata directory not provided
I20260620 04:29:01.677819 20104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:29:01.678025 20104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:29:01.679977 20104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929496330398-392-0/minicluster-data/master-0/data/instance:
uuid: "d54fdd13f580457eab5653a8c89e2078"
format_stamp: "Formatted at 2026-06-20 04:29:01 on dist-test-slave-nk6d"
server_key: "ee62a060ac66e1538a95d49b5715ef6e"
server_key_iv: "51db2cfd4a59a32a7b8283e679f5f05d"
server_key_version: "encryptionkey@0"
I20260620 04:29:01.680498 20104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929496330398-392-0/minicluster-data/master-0/wal/instance:
uuid: "d54fdd13f580457eab5653a8c89e2078"
format_stamp: "Formatted at 2026-06-20 04:29:01 on dist-test-slave-nk6d"
server_key: "ee62a060ac66e1538a95d49b5715ef6e"
server_key_iv: "51db2cfd4a59a32a7b8283e679f5f05d"
server_key_version: "encryptionkey@0"
I20260620 04:29:01.683818 20104 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 04:29:01.686028 20118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:29:01.687139 20104 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:29:01.687251 20104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929496330398-392-0/minicluster-data/master-0/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929496330398-392-0/minicluster-data/master-0/wal
uuid: "d54fdd13f580457eab5653a8c89e2078"
format_stamp: "Formatted at 2026-06-20 04:29:01 on dist-test-slave-nk6d"
server_key: "ee62a060ac66e1538a95d49b5715ef6e"
server_key_iv: "51db2cfd4a59a32a7b8283e679f5f05d"
server_key_version: "encryptionkey@0"
I20260620 04:29:01.687341 20104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929496330398-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929496330398-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929496330398-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:29:01.711086 20104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:29:01.711822 20104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:29:01.712045 20104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:29:01.718696 20104 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42191
I20260620 04:29:01.718700 20170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42191 every 8 connection(s)
I20260620 04:29:01.719601 20104 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:29:01.722591 20171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:29:01.726596 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 20104
I20260620 04:29:01.726699 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929496330398-392-0/minicluster-data/master-0/wal/instance
I20260620 04:29:01.726970 392 external_mini_cluster.cc:1428] Setting key c4488a4a864ccb79a0bffeb17d3fc544
I20260620 04:29:01.728145 20171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d54fdd13f580457eab5653a8c89e2078: Bootstrap starting.
I20260620 04:29:01.730767 20171 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d54fdd13f580457eab5653a8c89e2078: Neither blocks nor log segments found. Creating new log.
I20260620 04:29:01.731629 20171 log.cc:824] T 00000000000000000000000000000000 P d54fdd13f580457eab5653a8c89e2078: Log is configured to *not* fsync() on all Append() calls
I20260620 04:29:01.733752 20171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d54fdd13f580457eab5653a8c89e2078: No bootstrap required, opened a new log
I20260620 04:29:01.737509 20174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:01.728377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58072 (local address 127.0.98.62:42191)
0620 04:29:01.729038 (+ 661us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:01.729055 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:01.729096 (+ 41us) server_negotiation.cc:407] Connection header received
0620 04:29:01.729921 (+ 825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:01.729944 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:01.730426 (+ 482us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:01.730786 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:29:01.731263 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:01.732572 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:01.733657 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:01.734325 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:01.734489 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:01.735262 (+ 773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:29:01.735269 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:29:01.735285 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:29:01.735859 (+ 574us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:29:01.735930 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:29:01.735979 (+ 49us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:29:01.736072 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 04:29:01.736220 (+ 148us) server_negotiation.cc:299] Negotiation successful
0620 04:29:01.736494 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":101,"threads_started":1}
I20260620 04:29:01.738237 20171 raft_consensus.cc:348] T 00000000000000000000000000000000 P d54fdd13f580457eab5653a8c89e2078 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d54fdd13f580457eab5653a8c89e2078" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42191 } }
I20260620 04:29:01.738533 20171 raft_consensus.cc:374] T 00000000000000000000000000000000 P d54fdd13f580457eab5653a8c89e2078 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:29:01.738598 20171 raft_consensus.cc:729] T 00000000000000000000000000000000 P d54fdd13f580457eab5653a8c89e2078 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d54fdd13f580457eab5653a8c89e2078, State: Initialized, Role: FOLLOWER
I20260620 04:29:01.739092 20171 consensus_queue.cc:260] T 00000000000000000000000000000000 P d54fdd13f580457eab5653a8c89e2078 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d54fdd13f580457eab5653a8c89e2078" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42191 } }
I20260620 04:29:01.739254 20171 raft_consensus.cc:388] T 00000000000000000000000000000000 P d54fdd13f580457eab5653a8c89e2078 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:29:01.739332 20171 raft_consensus.cc:482] T 00000000000000000000000000000000 P d54fdd13f580457eab5653a8c89e2078 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:29:01.739444 20171 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d54fdd13f580457eab5653a8c89e2078 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:29:01.740597 20171 raft_consensus.cc:504] T 00000000000000000000000000000000 P d54fdd13f580457eab5653a8c89e2078 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d54fdd13f580457eab5653a8c89e2078" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42191 } }
I20260620 04:29:01.740975 20171 leader_election.cc:302] T 00000000000000000000000000000000 P d54fdd13f580457eab5653a8c89e2078 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d54fdd13f580457eab5653a8c89e2078; no voters:
I20260620 04:29:01.741344 20171 leader_election.cc:288] T 00000000000000000000000000000000 P d54fdd13f580457eab5653a8c89e2078 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:29:01.741433 20176 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d54fdd13f580457eab5653a8c89e2078 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:29:01.741636 20176 raft_consensus.cc:686] T 00000000000000000000000000000000 P d54fdd13f580457eab5653a8c89e2078 [term 1 LEADER]: Becoming Leader. State: Replica: d54fdd13f580457eab5653a8c89e2078, State: Running, Role: LEADER
I20260620 04:29:01.741964 20176 consensus_queue.cc:237] T 00000000000000000000000000000000 P d54fdd13f580457eab5653a8c89e2078 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d54fdd13f580457eab5653a8c89e2078" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42191 } }
I20260620 04:29:01.742523 20171 sys_catalog.cc:564] T 00000000000000000000000000000000 P d54fdd13f580457eab5653a8c89e2078 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:29:01.743541 20177 sys_catalog.cc:455] T 00000000000000000000000000000000 P d54fdd13f580457eab5653a8c89e2078 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d54fdd13f580457eab5653a8c89e2078" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d54fdd13f580457eab5653a8c89e2078" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42191 } } }
I20260620 04:29:01.743660 20177 sys_catalog.cc:458] T 00000000000000000000000000000000 P d54fdd13f580457eab5653a8c89e2078 [sys.catalog]: This master's current role is: LEADER
I20260620 04:29:01.744065 20185 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:29:01.744360 20178 sys_catalog.cc:455] T 00000000000000000000000000000000 P d54fdd13f580457eab5653a8c89e2078 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d54fdd13f580457eab5653a8c89e2078. Latest consensus state: current_term: 1 leader_uuid: "d54fdd13f580457eab5653a8c89e2078" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d54fdd13f580457eab5653a8c89e2078" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42191 } } }
I20260620 04:29:01.744468 20178 sys_catalog.cc:458] T 00000000000000000000000000000000 P d54fdd13f580457eab5653a8c89e2078 [sys.catalog]: This master's current role is: LEADER
I20260620 04:29:01.747175 20185 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:29:01.753222 20185 catalog_manager.cc:1347] Generated new cluster ID: 812677fd6cdc47a7ab9741920274474c
I20260620 04:29:01.753294 20185 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:29:01.760716 20185 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:29:01.761881 20185 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:29:01.767086 20185 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d54fdd13f580457eab5653a8c89e2078: Generated new TSK 0
I20260620 04:29:01.767848 20185 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 04:29:01.776446 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929496330398-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929496330398-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929496330398-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929496330398-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42191
--builtin_ntp_servers=127.0.98.20:39543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 04:29:01.882995 20195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:29:01.883278 20195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:29:01.883335 20195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:29:01.887401 20195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:29:01.887466 20195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:29:01.887545 20195 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 04:29:01.892434 20195 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929496330398-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929496330398-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929496330398-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42191
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20195
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929496330398-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:29:01.893530 20195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:29:01.894887 20195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:29:01.904034 20203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:29:01.904028 20200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:29:01.904141 20195 server_base.cc:1034] running on GCE node
W20260620 04:29:01.904028 20201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:29:01.904649 20195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:29:01.905265 20195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:29:01.906425 20195 hybrid_clock.cc:648] HybridClock initialized: now 1781929741906405 us; error 32 us; skew 500 ppm
I20260620 04:29:01.908275 20195 webserver.cc:460] Webserver started at http://127.0.98.1:38253/ using document root <none> and password file <none>
I20260620 04:29:01.908833 20195 fs_manager.cc:359] Metadata directory not provided
I20260620 04:29:01.908897 20195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:29:01.909088 20195 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:29:01.911069 20195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929496330398-392-0/minicluster-data/ts-0/data/instance:
uuid: "8f858b533d01485694303c8788d6d2a7"
format_stamp: "Formatted at 2026-06-20 04:29:01 on dist-test-slave-nk6d"
server_key: "c02ea1d0ac90db5aaa9c90a7654bfc39"
server_key_iv: "60addcf56b34edeef2e48908e9c99d48"
server_key_version: "encryptionkey@0"
I20260620 04:29:01.911598 20195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929496330398-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8f858b533d01485694303c8788d6d2a7"
format_stamp: "Formatted at 2026-06-20 04:29:01 on dist-test-slave-nk6d"
server_key: "c02ea1d0ac90db5aaa9c90a7654bfc39"
server_key_iv: "60addcf56b34edeef2e48908e9c99d48"
server_key_version: "encryptionkey@0"
I20260620 04:29:01.915220 20195 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.001s
I20260620 04:29:01.917650 20209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:29:01.918848 20195 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 04:29:01.918953 20195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929496330398-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929496330398-392-0/minicluster-data/ts-0/wal
uuid: "8f858b533d01485694303c8788d6d2a7"
format_stamp: "Formatted at 2026-06-20 04:29:01 on dist-test-slave-nk6d"
server_key: "c02ea1d0ac90db5aaa9c90a7654bfc39"
server_key_iv: "60addcf56b34edeef2e48908e9c99d48"
server_key_version: "encryptionkey@0"
I20260620 04:29:01.919050 20195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929496330398-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929496330398-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929496330398-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:29:01.934329 20195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:29:01.935067 20195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:29:01.935249 20195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:29:01.935853 20195 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:29:01.936852 20195 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:29:01.936899 20195 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:29:01.936954 20195 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:29:01.936980 20195 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:29:01.945380 20195 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38135
I20260620 04:29:01.945441 20322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38135 every 8 connection(s)
I20260620 04:29:01.946344 20195 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929496330398-392-0/minicluster-data/ts-0/data/info.pb
I20260620 04:29:01.952159 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 20195
I20260620 04:29:01.952272 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929496330398-392-0/minicluster-data/ts-0/wal/instance
I20260620 04:29:01.952569 392 external_mini_cluster.cc:1428] Setting key ea048bfa86baf17080b6ba8d4f61d613
I20260620 04:29:01.954439 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929496330398-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929496330398-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929496330398-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929496330398-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42191
--builtin_ntp_servers=127.0.98.20:39543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 04:29:01.955551 20174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:01.948086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36237 (local address 127.0.98.62:42191)
0620 04:29:01.948223 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:01.948227 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:01.949313 (+ 1086us) server_negotiation.cc:407] Connection header received
0620 04:29:01.950752 (+ 1439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:01.950755 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:01.950812 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:01.950895 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:29:01.951799 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:01.952409 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:01.953760 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:01.954001 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:01.954085 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:01.955111 (+ 1026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:29:01.955115 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:29:01.955118 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:29:01.955131 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:29:01.955152 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:29:01.955160 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:29:01.955215 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 04:29:01.955350 (+ 135us) server_negotiation.cc:299] Negotiation successful
0620 04:29:01.955411 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 04:29:01.956437 20325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:01.948526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42191 (local address 127.0.98.1:36237)
0620 04:29:01.949171 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0620 04:29:01.949210 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 04:29:01.950534 (+ 1324us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:29:01.951077 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:29:01.951083 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:29:01.951170 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:29:01.951667 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:29:01.951680 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:29:01.952528 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:29:01.952530 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:29:01.953654 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:29:01.953665 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:29:01.954138 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:29:01.954142 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:29:01.954302 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:01.954866 (+ 564us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 04:29:01.954867 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:29:01.954906 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 04:29:01.954972 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:29:01.955225 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:29:01.955233 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:29:01.955239 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 04:29:01.955355 (+ 116us) client_negotiation.cc:240] Negotiation successful
0620 04:29:01.955638 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":106,"threads_started":1}
I20260620 04:29:01.957597 20323 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42191
I20260620 04:29:01.957830 20323 heartbeater.cc:464] Registering TS with master...
I20260620 04:29:01.958319 20323 heartbeater.cc:511] Master 127.0.98.62:42191 requested a full tablet report, sending...
I20260620 04:29:01.959769 20135 ts_manager.cc:194] Registered new tserver with Master: 8f858b533d01485694303c8788d6d2a7 (127.0.98.1:38135)
I20260620 04:29:01.960922 20135 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:36237
W20260620 04:29:02.061700 20326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:29:02.061968 20326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:29:02.062021 20326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:29:02.065989 20326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:29:02.066046 20326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:29:02.066120 20326 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 04:29:02.071079 20326 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929496330398-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929496330398-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929496330398-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42191
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20326
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929496330398-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:29:02.072189 20326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:29:02.073573 20326 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:29:02.082947 20331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:29:02.082937 20332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:29:02.083020 20326 server_base.cc:1034] running on GCE node
W20260620 04:29:02.082940 20334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:29:02.083570 20326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:29:02.084259 20326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:29:02.085438 20326 hybrid_clock.cc:648] HybridClock initialized: now 1781929742085411 us; error 37 us; skew 500 ppm
I20260620 04:29:02.087288 20326 webserver.cc:460] Webserver started at http://127.0.98.2:36485/ using document root <none> and password file <none>
I20260620 04:29:02.087847 20326 fs_manager.cc:359] Metadata directory not provided
I20260620 04:29:02.087908 20326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:29:02.088124 20326 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:29:02.090166 20326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929496330398-392-0/minicluster-data/ts-1/data/instance:
uuid: "b0734d5da4c5446cb5087ba3554fefbe"
format_stamp: "Formatted at 2026-06-20 04:29:02 on dist-test-slave-nk6d"
server_key: "1398c1127a27e57b74245326a02d222b"
server_key_iv: "d96e109a64542aab8e81f0b070d781da"
server_key_version: "encryptionkey@0"
I20260620 04:29:02.090714 20326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929496330398-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b0734d5da4c5446cb5087ba3554fefbe"
format_stamp: "Formatted at 2026-06-20 04:29:02 on dist-test-slave-nk6d"
server_key: "1398c1127a27e57b74245326a02d222b"
server_key_iv: "d96e109a64542aab8e81f0b070d781da"
server_key_version: "encryptionkey@0"
I20260620 04:29:02.094527 20326 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260620 04:29:02.097097 20340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:29:02.098335 20326 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:29:02.098445 20326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929496330398-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929496330398-392-0/minicluster-data/ts-1/wal
uuid: "b0734d5da4c5446cb5087ba3554fefbe"
format_stamp: "Formatted at 2026-06-20 04:29:02 on dist-test-slave-nk6d"
server_key: "1398c1127a27e57b74245326a02d222b"
server_key_iv: "d96e109a64542aab8e81f0b070d781da"
server_key_version: "encryptionkey@0"
I20260620 04:29:02.098546 20326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929496330398-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929496330398-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929496330398-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:29:02.113507 20326 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:29:02.114210 20326 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:29:02.114387 20326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:29:02.114959 20326 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:29:02.115903 20326 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:29:02.115948 20326 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:29:02.116061 20326 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:29:02.116093 20326 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:29:02.124812 20326 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35551
I20260620 04:29:02.124898 20453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35551 every 8 connection(s)
I20260620 04:29:02.125733 20326 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929496330398-392-0/minicluster-data/ts-1/data/info.pb
I20260620 04:29:02.130492 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 20326
I20260620 04:29:02.130620 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929496330398-392-0/minicluster-data/ts-1/wal/instance
I20260620 04:29:02.130941 392 external_mini_cluster.cc:1428] Setting key 39b2eb38500dcf515e0e790c8a070801
I20260620 04:29:02.133085 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929496330398-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929496330398-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929496330398-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929496330398-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42191
--builtin_ntp_servers=127.0.98.20:39543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 04:29:02.136045 20174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:02.127537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37789 (local address 127.0.98.62:42191)
0620 04:29:02.127679 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:02.127683 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:02.128750 (+ 1067us) server_negotiation.cc:407] Connection header received
0620 04:29:02.130168 (+ 1418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:02.130171 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:02.130221 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:02.130346 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:29:02.131567 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:02.132145 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:02.133770 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:02.134074 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:02.134188 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:02.135395 (+ 1207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:29:02.135400 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:29:02.135403 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:29:02.135420 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:29:02.135447 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:29:02.135460 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:29:02.135549 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 04:29:02.135778 (+ 229us) server_negotiation.cc:299] Negotiation successful
0620 04:29:02.135846 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 04:29:02.136989 20456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:02.127933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42191 (local address 127.0.98.2:37789)
0620 04:29:02.128617 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0620 04:29:02.128658 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 04:29:02.129918 (+ 1260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:29:02.130563 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:29:02.130573 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:29:02.130687 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:29:02.131401 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:29:02.131420 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:29:02.132264 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:29:02.132268 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:29:02.133597 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:29:02.133609 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:29:02.134290 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:29:02.134297 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:29:02.134512 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:02.135105 (+ 593us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 04:29:02.135106 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:29:02.135150 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 04:29:02.135221 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:29:02.135613 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:29:02.135622 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:29:02.135628 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 04:29:02.135754 (+ 126us) client_negotiation.cc:240] Negotiation successful
0620 04:29:02.136072 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":100,"threads_started":1}
I20260620 04:29:02.138178 20454 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42191
I20260620 04:29:02.138391 20454 heartbeater.cc:464] Registering TS with master...
I20260620 04:29:02.138921 20454 heartbeater.cc:511] Master 127.0.98.62:42191 requested a full tablet report, sending...
I20260620 04:29:02.139894 20135 ts_manager.cc:194] Registered new tserver with Master: b0734d5da4c5446cb5087ba3554fefbe (127.0.98.2:35551)
I20260620 04:29:02.140486 20135 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:37789
W20260620 04:29:02.242750 20457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:29:02.243032 20457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:29:02.243116 20457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:29:02.247248 20457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:29:02.247308 20457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:29:02.247381 20457 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 04:29:02.252070 20457 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929496330398-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929496330398-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929496330398-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42191
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20457
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929496330398-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:29:02.253206 20457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:29:02.254606 20457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:29:02.263661 20463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:29:02.263651 20462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:29:02.263723 20457 server_base.cc:1034] running on GCE node
W20260620 04:29:02.263659 20465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:29:02.264317 20457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:29:02.264972 20457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:29:02.266126 20457 hybrid_clock.cc:648] HybridClock initialized: now 1781929742266109 us; error 30 us; skew 500 ppm
I20260620 04:29:02.267897 20457 webserver.cc:460] Webserver started at http://127.0.98.3:43241/ using document root <none> and password file <none>
I20260620 04:29:02.268481 20457 fs_manager.cc:359] Metadata directory not provided
I20260620 04:29:02.268558 20457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:29:02.268765 20457 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:29:02.270740 20457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929496330398-392-0/minicluster-data/ts-2/data/instance:
uuid: "3f0b35a9cc30436591f8fcf99f08c91e"
format_stamp: "Formatted at 2026-06-20 04:29:02 on dist-test-slave-nk6d"
server_key: "5ebd5fd2a1786fa9d200c704d4141394"
server_key_iv: "6696a6f8082b4f1325a5714099358894"
server_key_version: "encryptionkey@0"
I20260620 04:29:02.271250 20457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929496330398-392-0/minicluster-data/ts-2/wal/instance:
uuid: "3f0b35a9cc30436591f8fcf99f08c91e"
format_stamp: "Formatted at 2026-06-20 04:29:02 on dist-test-slave-nk6d"
server_key: "5ebd5fd2a1786fa9d200c704d4141394"
server_key_iv: "6696a6f8082b4f1325a5714099358894"
server_key_version: "encryptionkey@0"
I20260620 04:29:02.274807 20457 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260620 04:29:02.277104 20471 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:29:02.278263 20457 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 04:29:02.278350 20457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929496330398-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929496330398-392-0/minicluster-data/ts-2/wal
uuid: "3f0b35a9cc30436591f8fcf99f08c91e"
format_stamp: "Formatted at 2026-06-20 04:29:02 on dist-test-slave-nk6d"
server_key: "5ebd5fd2a1786fa9d200c704d4141394"
server_key_iv: "6696a6f8082b4f1325a5714099358894"
server_key_version: "encryptionkey@0"
I20260620 04:29:02.278472 20457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929496330398-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929496330398-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929496330398-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:29:02.292479 20457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:29:02.293136 20457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:29:02.293283 20457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:29:02.293865 20457 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:29:02.294966 20457 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:29:02.295011 20457 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:29:02.295053 20457 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:29:02.295078 20457 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:29:02.303081 20457 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39963
I20260620 04:29:02.303130 20584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39963 every 8 connection(s)
I20260620 04:29:02.304000 20457 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929496330398-392-0/minicluster-data/ts-2/data/info.pb
I20260620 04:29:02.309427 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 20457
I20260620 04:29:02.309547 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929496330398-392-0/minicluster-data/ts-2/wal/instance
I20260620 04:29:02.309821 392 external_mini_cluster.cc:1428] Setting key 749775f88b524583f82aed2efe3e39be
I20260620 04:29:02.313321 20174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:02.305750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41185 (local address 127.0.98.62:42191)
0620 04:29:02.305873 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:02.305877 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:02.306914 (+ 1037us) server_negotiation.cc:407] Connection header received
0620 04:29:02.308280 (+ 1366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:02.308283 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:02.308328 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:02.308405 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:29:02.309288 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:02.310064 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:02.311558 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:02.311769 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:02.311854 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:02.312919 (+ 1065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:29:02.312922 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:29:02.312925 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:29:02.312938 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:29:02.312960 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:29:02.312969 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:29:02.313053 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 04:29:02.313165 (+ 112us) server_negotiation.cc:299] Negotiation successful
0620 04:29:02.313213 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 04:29:02.314193 20587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:02.306120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42191 (local address 127.0.98.3:41185)
0620 04:29:02.306759 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0620 04:29:02.306800 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 04:29:02.308105 (+ 1305us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:29:02.308545 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:29:02.308552 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:29:02.308632 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:29:02.309153 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:29:02.309165 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:29:02.310189 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:29:02.310192 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:29:02.311463 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:29:02.311474 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:29:02.311916 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:29:02.311919 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:29:02.312114 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:02.312680 (+ 566us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 04:29:02.312682 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:29:02.312722 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 04:29:02.312785 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:29:02.313057 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:29:02.313064 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:29:02.313069 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 04:29:02.313161 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 04:29:02.313387 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":98,"threads_started":1}
I20260620 04:29:02.315168 20585 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42191
I20260620 04:29:02.315382 20585 heartbeater.cc:464] Registering TS with master...
I20260620 04:29:02.315860 20585 heartbeater.cc:511] Master 127.0.98.62:42191 requested a full tablet report, sending...
I20260620 04:29:02.316855 20135 ts_manager.cc:194] Registered new tserver with Master: 3f0b35a9cc30436591f8fcf99f08c91e (127.0.98.3:39963)
I20260620 04:29:02.317366 20135 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:41185
I20260620 04:29:02.323661 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 04:29:02.329242 20174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:02.325613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58074 (local address 127.0.98.62:42191)
0620 04:29:02.325761 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:02.325765 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:02.325878 (+ 113us) server_negotiation.cc:407] Connection header received
0620 04:29:02.325979 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:02.325982 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:02.326028 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:02.326093 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:29:02.326354 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:02.326877 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:02.327733 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:02.328004 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:02.328093 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:02.328862 (+ 769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:29:02.328866 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:29:02.328869 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:29:02.328883 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:29:02.328900 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:29:02.328907 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:29:02.328964 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 04:29:02.329047 (+ 83us) server_negotiation.cc:299] Negotiation successful
0620 04:29:02.329102 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 04:29:02.331851 20135 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58074:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 04:29:02.334894 20135 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 04:29:02.348433 20596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:02.343037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39963 (local address 127.0.0.1:57168)
0620 04:29:02.343546 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0620 04:29:02.343596 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 04:29:02.343716 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:29:02.344862 (+ 1146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:29:02.344865 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:29:02.344877 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:29:02.344958 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:29:02.344962 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:29:02.345868 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:29:02.345872 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:29:02.346922 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:29:02.346930 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:29:02.347888 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:29:02.347891 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:29:02.348021 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:02.348059 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 04:29:02.348160 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 04:29:02.348273 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"mutex_wait_us":51,"thread_start_us":83,"threads_started":1}
I20260620 04:29:02.348438 20601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:02.344021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57168 (local address 127.0.98.3:39963)
0620 04:29:02.344444 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:02.344459 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:02.344480 (+ 21us) server_negotiation.cc:407] Connection header received
0620 04:29:02.344548 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:02.344552 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:02.344715 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:02.344890 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:29:02.345027 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:02.345764 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:02.347028 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:02.347781 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:02.347875 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:02.348107 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0620 04:29:02.348185 (+ 78us) server_negotiation.cc:299] Negotiation successful
0620 04:29:02.348273 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":138,"threads_started":1}
I20260620 04:29:02.349038 20599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:02.343157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38135 (local address 127.0.0.1:34284)
0620 04:29:02.344047 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0620 04:29:02.344059 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 04:29:02.344157 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:29:02.344723 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:29:02.344729 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:29:02.344749 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:29:02.344884 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:29:02.344892 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:29:02.345873 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:29:02.345877 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:29:02.347100 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:29:02.347110 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:29:02.348495 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:29:02.348498 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:29:02.348580 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:02.348602 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 04:29:02.348843 (+ 241us) client_negotiation.cc:240] Negotiation successful
0620 04:29:02.348907 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":830,"mutex_wait_us":31,"thread_start_us":77,"threads_started":1}
I20260620 04:29:02.349540 20598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:02.343319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34284 (local address 127.0.98.1:38135)
0620 04:29:02.343820 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:02.343833 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:02.344160 (+ 327us) server_negotiation.cc:407] Connection header received
0620 04:29:02.344251 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:02.344256 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:02.344477 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:02.344623 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:29:02.345015 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:02.345763 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:02.347358 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:02.348160 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:02.348258 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:02.348436 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0620 04:29:02.349290 (+ 854us) server_negotiation.cc:299] Negotiation successful
0620 04:29:02.349374 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":117,"threads_started":1}
I20260620 04:29:02.350314 20600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:02.343988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54196 (local address 127.0.98.2:35551)
0620 04:29:02.346041 (+ 2053us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:02.346052 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:02.346075 (+ 23us) server_negotiation.cc:407] Connection header received
0620 04:29:02.346144 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:02.346149 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:02.346324 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:02.346464 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:29:02.346673 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:02.347711 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:02.349084 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:02.349780 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:02.349875 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:02.350053 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0620 04:29:02.350097 (+ 44us) server_negotiation.cc:299] Negotiation successful
0620 04:29:02.350178 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1947,"thread_start_us":107,"threads_started":1}
I20260620 04:29:02.350617 20597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:02.343773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35551 (local address 127.0.0.1:54196)
0620 04:29:02.344520 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0620 04:29:02.344532 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 04:29:02.344608 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:29:02.346462 (+ 1854us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:29:02.346465 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:29:02.346484 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:29:02.346572 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:29:02.346577 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:29:02.347824 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:29:02.347827 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:29:02.348968 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:29:02.348977 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:29:02.349897 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:29:02.349900 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:29:02.349980 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:02.350001 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 04:29:02.350058 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 04:29:02.350119 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":685,"thread_start_us":103,"threads_started":1}
I20260620 04:29:02.351735 20519 tablet_service.cc:1467] Processing CreateTablet for tablet 068238cff6cf447cb19adfe4cd35c47c (DEFAULT_TABLE table=test-table [id=a19b8f580ac34407b8dbbe055972f7c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:29:02.352210 20257 tablet_service.cc:1467] Processing CreateTablet for tablet 068238cff6cf447cb19adfe4cd35c47c (DEFAULT_TABLE table=test-table [id=a19b8f580ac34407b8dbbe055972f7c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:29:02.352938 20519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 068238cff6cf447cb19adfe4cd35c47c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:29:02.353191 20257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 068238cff6cf447cb19adfe4cd35c47c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:29:02.354377 20388 tablet_service.cc:1467] Processing CreateTablet for tablet 068238cff6cf447cb19adfe4cd35c47c (DEFAULT_TABLE table=test-table [id=a19b8f580ac34407b8dbbe055972f7c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:29:02.355155 20388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 068238cff6cf447cb19adfe4cd35c47c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:29:02.361680 20603 tablet_bootstrap.cc:492] T 068238cff6cf447cb19adfe4cd35c47c P 3f0b35a9cc30436591f8fcf99f08c91e: Bootstrap starting.
I20260620 04:29:02.363168 20602 tablet_bootstrap.cc:492] T 068238cff6cf447cb19adfe4cd35c47c P 8f858b533d01485694303c8788d6d2a7: Bootstrap starting.
I20260620 04:29:02.363993 20603 tablet_bootstrap.cc:654] T 068238cff6cf447cb19adfe4cd35c47c P 3f0b35a9cc30436591f8fcf99f08c91e: Neither blocks nor log segments found. Creating new log.
I20260620 04:29:02.364813 20603 log.cc:824] T 068238cff6cf447cb19adfe4cd35c47c P 3f0b35a9cc30436591f8fcf99f08c91e: Log is configured to *not* fsync() on all Append() calls
I20260620 04:29:02.365372 20602 tablet_bootstrap.cc:654] T 068238cff6cf447cb19adfe4cd35c47c P 8f858b533d01485694303c8788d6d2a7: Neither blocks nor log segments found. Creating new log.
I20260620 04:29:02.366006 20602 log.cc:824] T 068238cff6cf447cb19adfe4cd35c47c P 8f858b533d01485694303c8788d6d2a7: Log is configured to *not* fsync() on all Append() calls
I20260620 04:29:02.366580 20604 tablet_bootstrap.cc:492] T 068238cff6cf447cb19adfe4cd35c47c P b0734d5da4c5446cb5087ba3554fefbe: Bootstrap starting.
I20260620 04:29:02.366637 20603 tablet_bootstrap.cc:492] T 068238cff6cf447cb19adfe4cd35c47c P 3f0b35a9cc30436591f8fcf99f08c91e: No bootstrap required, opened a new log
I20260620 04:29:02.366865 20603 ts_tablet_manager.cc:1397] T 068238cff6cf447cb19adfe4cd35c47c P 3f0b35a9cc30436591f8fcf99f08c91e: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 04:29:02.368294 20604 tablet_bootstrap.cc:654] T 068238cff6cf447cb19adfe4cd35c47c P b0734d5da4c5446cb5087ba3554fefbe: Neither blocks nor log segments found. Creating new log.
I20260620 04:29:02.368389 20602 tablet_bootstrap.cc:492] T 068238cff6cf447cb19adfe4cd35c47c P 8f858b533d01485694303c8788d6d2a7: No bootstrap required, opened a new log
I20260620 04:29:02.368582 20602 ts_tablet_manager.cc:1397] T 068238cff6cf447cb19adfe4cd35c47c P 8f858b533d01485694303c8788d6d2a7: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20260620 04:29:02.369074 20604 log.cc:824] T 068238cff6cf447cb19adfe4cd35c47c P b0734d5da4c5446cb5087ba3554fefbe: Log is configured to *not* fsync() on all Append() calls
I20260620 04:29:02.370997 20603 raft_consensus.cc:348] T 068238cff6cf447cb19adfe4cd35c47c P 3f0b35a9cc30436591f8fcf99f08c91e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f0b35a9cc30436591f8fcf99f08c91e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39963 } } peers { permanent_uuid: "8f858b533d01485694303c8788d6d2a7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38135 } } peers { permanent_uuid: "b0734d5da4c5446cb5087ba3554fefbe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35551 } }
I20260620 04:29:02.371219 20604 tablet_bootstrap.cc:492] T 068238cff6cf447cb19adfe4cd35c47c P b0734d5da4c5446cb5087ba3554fefbe: No bootstrap required, opened a new log
I20260620 04:29:02.371279 20603 raft_consensus.cc:374] T 068238cff6cf447cb19adfe4cd35c47c P 3f0b35a9cc30436591f8fcf99f08c91e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:29:02.371332 20603 raft_consensus.cc:729] T 068238cff6cf447cb19adfe4cd35c47c P 3f0b35a9cc30436591f8fcf99f08c91e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f0b35a9cc30436591f8fcf99f08c91e, State: Initialized, Role: FOLLOWER
I20260620 04:29:02.371397 20604 ts_tablet_manager.cc:1397] T 068238cff6cf447cb19adfe4cd35c47c P b0734d5da4c5446cb5087ba3554fefbe: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 04:29:02.371801 20603 consensus_queue.cc:260] T 068238cff6cf447cb19adfe4cd35c47c P 3f0b35a9cc30436591f8fcf99f08c91e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f0b35a9cc30436591f8fcf99f08c91e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39963 } } peers { permanent_uuid: "8f858b533d01485694303c8788d6d2a7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38135 } } peers { permanent_uuid: "b0734d5da4c5446cb5087ba3554fefbe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35551 } }
I20260620 04:29:02.372509 20585 heartbeater.cc:503] Master 127.0.98.62:42191 was elected leader, sending a full tablet report...
I20260620 04:29:02.372820 20603 ts_tablet_manager.cc:1428] T 068238cff6cf447cb19adfe4cd35c47c P 3f0b35a9cc30436591f8fcf99f08c91e: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 04:29:02.372941 20602 raft_consensus.cc:348] T 068238cff6cf447cb19adfe4cd35c47c P 8f858b533d01485694303c8788d6d2a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f0b35a9cc30436591f8fcf99f08c91e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39963 } } peers { permanent_uuid: "8f858b533d01485694303c8788d6d2a7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38135 } } peers { permanent_uuid: "b0734d5da4c5446cb5087ba3554fefbe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35551 } }
I20260620 04:29:02.373195 20602 raft_consensus.cc:374] T 068238cff6cf447cb19adfe4cd35c47c P 8f858b533d01485694303c8788d6d2a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:29:02.373250 20602 raft_consensus.cc:729] T 068238cff6cf447cb19adfe4cd35c47c P 8f858b533d01485694303c8788d6d2a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f858b533d01485694303c8788d6d2a7, State: Initialized, Role: FOLLOWER
I20260620 04:29:02.373677 20602 consensus_queue.cc:260] T 068238cff6cf447cb19adfe4cd35c47c P 8f858b533d01485694303c8788d6d2a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f0b35a9cc30436591f8fcf99f08c91e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39963 } } peers { permanent_uuid: "8f858b533d01485694303c8788d6d2a7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38135 } } peers { permanent_uuid: "b0734d5da4c5446cb5087ba3554fefbe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35551 } }
I20260620 04:29:02.374320 20323 heartbeater.cc:503] Master 127.0.98.62:42191 was elected leader, sending a full tablet report...
I20260620 04:29:02.374625 20602 ts_tablet_manager.cc:1428] T 068238cff6cf447cb19adfe4cd35c47c P 8f858b533d01485694303c8788d6d2a7: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260620 04:29:02.375269 20604 raft_consensus.cc:348] T 068238cff6cf447cb19adfe4cd35c47c P b0734d5da4c5446cb5087ba3554fefbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f0b35a9cc30436591f8fcf99f08c91e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39963 } } peers { permanent_uuid: "8f858b533d01485694303c8788d6d2a7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38135 } } peers { permanent_uuid: "b0734d5da4c5446cb5087ba3554fefbe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35551 } }
I20260620 04:29:02.375813 20604 raft_consensus.cc:374] T 068238cff6cf447cb19adfe4cd35c47c P b0734d5da4c5446cb5087ba3554fefbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:29:02.375903 20604 raft_consensus.cc:729] T 068238cff6cf447cb19adfe4cd35c47c P b0734d5da4c5446cb5087ba3554fefbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0734d5da4c5446cb5087ba3554fefbe, State: Initialized, Role: FOLLOWER
I20260620 04:29:02.376350 20604 consensus_queue.cc:260] T 068238cff6cf447cb19adfe4cd35c47c P b0734d5da4c5446cb5087ba3554fefbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f0b35a9cc30436591f8fcf99f08c91e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39963 } } peers { permanent_uuid: "8f858b533d01485694303c8788d6d2a7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38135 } } peers { permanent_uuid: "b0734d5da4c5446cb5087ba3554fefbe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35551 } }
I20260620 04:29:02.376945 20454 heartbeater.cc:503] Master 127.0.98.62:42191 was elected leader, sending a full tablet report...
I20260620 04:29:02.377243 20604 ts_tablet_manager.cc:1428] T 068238cff6cf447cb19adfe4cd35c47c P b0734d5da4c5446cb5087ba3554fefbe: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260620 04:29:02.447481 20324 tablet.cc:2369] T 068238cff6cf447cb19adfe4cd35c47c P 8f858b533d01485694303c8788d6d2a7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 04:29:02.554883 20586 tablet.cc:2369] T 068238cff6cf447cb19adfe4cd35c47c P 3f0b35a9cc30436591f8fcf99f08c91e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 04:29:02.578502 20610 raft_consensus.cc:482] T 068238cff6cf447cb19adfe4cd35c47c P b0734d5da4c5446cb5087ba3554fefbe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:29:02.578730 20610 raft_consensus.cc:504] T 068238cff6cf447cb19adfe4cd35c47c P b0734d5da4c5446cb5087ba3554fefbe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f0b35a9cc30436591f8fcf99f08c91e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39963 } } peers { permanent_uuid: "8f858b533d01485694303c8788d6d2a7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38135 } } peers { permanent_uuid: "b0734d5da4c5446cb5087ba3554fefbe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35551 } }
I20260620 04:29:02.580005 20610 leader_election.cc:288] T 068238cff6cf447cb19adfe4cd35c47c P b0734d5da4c5446cb5087ba3554fefbe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f0b35a9cc30436591f8fcf99f08c91e (127.0.98.3:39963), 8f858b533d01485694303c8788d6d2a7 (127.0.98.1:38135)
I20260620 04:29:02.584662 20601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:02.580325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48647 (local address 127.0.98.3:39963)
0620 04:29:02.580559 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:02.580564 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:02.580579 (+ 15us) server_negotiation.cc:407] Connection header received
0620 04:29:02.580637 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:02.580640 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:02.580683 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:02.580750 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:29:02.581097 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:02.581800 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:02.583625 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:02.584105 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:02.584189 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:02.584240 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 04:29:02.584490 (+ 250us) server_negotiation.cc:299] Negotiation successful
0620 04:29:02.584555 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160}
I20260620 04:29:02.584707 20456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:02.580182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39963 (local address 127.0.98.2:48647)
0620 04:29:02.580355 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 04:29:02.580370 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 04:29:02.580495 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:29:02.580794 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:29:02.580797 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:29:02.580817 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:29:02.580960 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:29:02.580968 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:29:02.581936 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:29:02.581940 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:29:02.583484 (+ 1544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:29:02.583496 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:29:02.584239 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:29:02.584243 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:29:02.584357 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:02.584387 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 04:29:02.584473 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 04:29:02.584554 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"mutex_wait_us":134}
I20260620 04:29:02.585345 20611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:02.580220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38135 (local address 127.0.98.2:36605)
0620 04:29:02.580555 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0620 04:29:02.580568 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:29:02.580668 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:29:02.580999 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:29:02.581002 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:29:02.581023 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:29:02.581173 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:29:02.581181 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:29:02.581993 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:29:02.581996 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:29:02.583484 (+ 1488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:29:02.583496 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:29:02.584938 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:29:02.584942 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:29:02.585046 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:02.585069 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 04:29:02.585140 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 04:29:02.585209 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"mutex_wait_us":102,"thread_start_us":129,"threads_started":1}
I20260620 04:29:02.585361 20598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:02.580375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36605 (local address 127.0.98.1:38135)
0620 04:29:02.580506 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:02.580511 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:02.580669 (+ 158us) server_negotiation.cc:407] Connection header received
0620 04:29:02.580809 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:02.580813 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:02.580864 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:02.580944 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:29:02.581346 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:02.581899 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:02.584084 (+ 2185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:02.584696 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:02.584812 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:02.584878 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 04:29:02.585174 (+ 296us) server_negotiation.cc:299] Negotiation successful
0620 04:29:02.585257 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 04:29:02.585407 20539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "068238cff6cf447cb19adfe4cd35c47c" candidate_uuid: "b0734d5da4c5446cb5087ba3554fefbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f0b35a9cc30436591f8fcf99f08c91e" is_pre_election: true
I20260620 04:29:02.585755 20539 raft_consensus.cc:2454] T 068238cff6cf447cb19adfe4cd35c47c P 3f0b35a9cc30436591f8fcf99f08c91e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0734d5da4c5446cb5087ba3554fefbe in term 0.
I20260620 04:29:02.586089 20277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "068238cff6cf447cb19adfe4cd35c47c" candidate_uuid: "b0734d5da4c5446cb5087ba3554fefbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f858b533d01485694303c8788d6d2a7" is_pre_election: true
I20260620 04:29:02.586295 20342 leader_election.cc:302] T 068238cff6cf447cb19adfe4cd35c47c P b0734d5da4c5446cb5087ba3554fefbe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f0b35a9cc30436591f8fcf99f08c91e, b0734d5da4c5446cb5087ba3554fefbe; no voters:
I20260620 04:29:02.586342 20277 raft_consensus.cc:2454] T 068238cff6cf447cb19adfe4cd35c47c P 8f858b533d01485694303c8788d6d2a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0734d5da4c5446cb5087ba3554fefbe in term 0.
I20260620 04:29:02.586586 20610 raft_consensus.cc:2781] T 068238cff6cf447cb19adfe4cd35c47c P b0734d5da4c5446cb5087ba3554fefbe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 04:29:02.586684 20610 raft_consensus.cc:482] T 068238cff6cf447cb19adfe4cd35c47c P b0734d5da4c5446cb5087ba3554fefbe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 04:29:02.586738 20610 raft_consensus.cc:3037] T 068238cff6cf447cb19adfe4cd35c47c P b0734d5da4c5446cb5087ba3554fefbe [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:29:02.587718 20610 raft_consensus.cc:504] T 068238cff6cf447cb19adfe4cd35c47c P b0734d5da4c5446cb5087ba3554fefbe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f0b35a9cc30436591f8fcf99f08c91e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39963 } } peers { permanent_uuid: "8f858b533d01485694303c8788d6d2a7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38135 } } peers { permanent_uuid: "b0734d5da4c5446cb5087ba3554fefbe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35551 } }
I20260620 04:29:02.588053 20610 leader_election.cc:288] T 068238cff6cf447cb19adfe4cd35c47c P b0734d5da4c5446cb5087ba3554fefbe [CANDIDATE]: Term 1 election: Requested vote from peers 3f0b35a9cc30436591f8fcf99f08c91e (127.0.98.3:39963), 8f858b533d01485694303c8788d6d2a7 (127.0.98.1:38135)
I20260620 04:29:02.588438 20277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "068238cff6cf447cb19adfe4cd35c47c" candidate_uuid: "b0734d5da4c5446cb5087ba3554fefbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f858b533d01485694303c8788d6d2a7"
I20260620 04:29:02.588441 20539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "068238cff6cf447cb19adfe4cd35c47c" candidate_uuid: "b0734d5da4c5446cb5087ba3554fefbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f0b35a9cc30436591f8fcf99f08c91e"
I20260620 04:29:02.588552 20277 raft_consensus.cc:3037] T 068238cff6cf447cb19adfe4cd35c47c P 8f858b533d01485694303c8788d6d2a7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:29:02.588548 20539 raft_consensus.cc:3037] T 068238cff6cf447cb19adfe4cd35c47c P 3f0b35a9cc30436591f8fcf99f08c91e [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:29:02.589535 20277 raft_consensus.cc:2454] T 068238cff6cf447cb19adfe4cd35c47c P 8f858b533d01485694303c8788d6d2a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0734d5da4c5446cb5087ba3554fefbe in term 1.
I20260620 04:29:02.589612 20539 raft_consensus.cc:2454] T 068238cff6cf447cb19adfe4cd35c47c P 3f0b35a9cc30436591f8fcf99f08c91e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0734d5da4c5446cb5087ba3554fefbe in term 1.
I20260620 04:29:02.589818 20344 leader_election.cc:302] T 068238cff6cf447cb19adfe4cd35c47c P b0734d5da4c5446cb5087ba3554fefbe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f858b533d01485694303c8788d6d2a7, b0734d5da4c5446cb5087ba3554fefbe; no voters:
I20260620 04:29:02.589994 20610 raft_consensus.cc:2781] T 068238cff6cf447cb19adfe4cd35c47c P b0734d5da4c5446cb5087ba3554fefbe [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:29:02.590210 20610 raft_consensus.cc:686] T 068238cff6cf447cb19adfe4cd35c47c P b0734d5da4c5446cb5087ba3554fefbe [term 1 LEADER]: Becoming Leader. State: Replica: b0734d5da4c5446cb5087ba3554fefbe, State: Running, Role: LEADER
I20260620 04:29:02.590567 20610 consensus_queue.cc:237] T 068238cff6cf447cb19adfe4cd35c47c P b0734d5da4c5446cb5087ba3554fefbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f0b35a9cc30436591f8fcf99f08c91e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39963 } } peers { permanent_uuid: "8f858b533d01485694303c8788d6d2a7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38135 } } peers { permanent_uuid: "b0734d5da4c5446cb5087ba3554fefbe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35551 } }
I20260620 04:29:02.593545 20135 catalog_manager.cc:5495] T 068238cff6cf447cb19adfe4cd35c47c P b0734d5da4c5446cb5087ba3554fefbe reported cstate change: term changed from 0 to 1, leader changed from <none> to b0734d5da4c5446cb5087ba3554fefbe (127.0.98.2). New cstate: current_term: 1 leader_uuid: "b0734d5da4c5446cb5087ba3554fefbe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f0b35a9cc30436591f8fcf99f08c91e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39963 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f858b533d01485694303c8788d6d2a7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38135 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b0734d5da4c5446cb5087ba3554fefbe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35551 } health_report { overall_health: HEALTHY } } }
W20260620 04:29:02.626796 20455 tablet.cc:2369] T 068238cff6cf447cb19adfe4cd35c47c P b0734d5da4c5446cb5087ba3554fefbe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 04:29:02.644857 20600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:02.641184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54202 (local address 127.0.98.2:35551)
0620 04:29:02.641362 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:02.641368 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:02.641382 (+ 14us) server_negotiation.cc:407] Connection header received
0620 04:29:02.641459 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:02.641461 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:02.641515 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:02.641618 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:29:02.641896 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:02.642487 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:02.643378 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:02.643636 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:02.643725 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:02.644035 (+ 310us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:29:02.644465 (+ 430us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:29:02.644575 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 04:29:02.644663 (+ 88us) server_negotiation.cc:299] Negotiation successful
0620 04:29:02.644717 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 04:29:02.650475 20277 raft_consensus.cc:1261] T 068238cff6cf447cb19adfe4cd35c47c P 8f858b533d01485694303c8788d6d2a7 [term 1 FOLLOWER]: Refusing update from remote peer b0734d5da4c5446cb5087ba3554fefbe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:29:02.650473 20539 raft_consensus.cc:1261] T 068238cff6cf447cb19adfe4cd35c47c P 3f0b35a9cc30436591f8fcf99f08c91e [term 1 FOLLOWER]: Refusing update from remote peer b0734d5da4c5446cb5087ba3554fefbe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:29:02.651160 20612 consensus_queue.cc:1035] T 068238cff6cf447cb19adfe4cd35c47c P b0734d5da4c5446cb5087ba3554fefbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f0b35a9cc30436591f8fcf99f08c91e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:29:02.651362 20610 consensus_queue.cc:1035] T 068238cff6cf447cb19adfe4cd35c47c P b0734d5da4c5446cb5087ba3554fefbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f858b533d01485694303c8788d6d2a7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:29:02.658550 20615 mvcc.cc:205] Tried to move back new op lower bound from 7298784225889529856 to 7298784225653567488. Current Snapshot: MvccSnapshot[applied={T|T < 7298784225889529856}]
I20260620 04:29:02.666757 20133 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58074:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:29:02.666913 20133 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58074:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:29:02.669154 20133 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d54fdd13f580457eab5653a8c89e2078: Sending DeleteTablet for 3 replicas of tablet 068238cff6cf447cb19adfe4cd35c47c
I20260620 04:29:02.669903 20519 tablet_service.cc:1514] Processing DeleteTablet for tablet 068238cff6cf447cb19adfe4cd35c47c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:29:02 UTC) from {username='slave'} at 127.0.0.1:57168
I20260620 04:29:02.670073 20257 tablet_service.cc:1514] Processing DeleteTablet for tablet 068238cff6cf447cb19adfe4cd35c47c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:29:02 UTC) from {username='slave'} at 127.0.0.1:34284
I20260620 04:29:02.670076 20388 tablet_service.cc:1514] Processing DeleteTablet for tablet 068238cff6cf447cb19adfe4cd35c47c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:29:02 UTC) from {username='slave'} at 127.0.0.1:54196
I20260620 04:29:02.670429 20625 tablet_replica.cc:330] T 068238cff6cf447cb19adfe4cd35c47c P 3f0b35a9cc30436591f8fcf99f08c91e: stopping tablet replica
I20260620 04:29:02.670486 20627 tablet_replica.cc:330] T 068238cff6cf447cb19adfe4cd35c47c P b0734d5da4c5446cb5087ba3554fefbe: stopping tablet replica
I20260620 04:29:02.670763 20627 raft_consensus.cc:2229] T 068238cff6cf447cb19adfe4cd35c47c P b0734d5da4c5446cb5087ba3554fefbe [term 1 LEADER]: Raft consensus shutting down.
I20260620 04:29:02.670763 20625 raft_consensus.cc:2229] T 068238cff6cf447cb19adfe4cd35c47c P 3f0b35a9cc30436591f8fcf99f08c91e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:29:02.670854 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 20195
I20260620 04:29:02.671075 20627 raft_consensus.cc:2258] T 068238cff6cf447cb19adfe4cd35c47c P b0734d5da4c5446cb5087ba3554fefbe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:29:02.671895 20625 raft_consensus.cc:2258] T 068238cff6cf447cb19adfe4cd35c47c P 3f0b35a9cc30436591f8fcf99f08c91e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:29:02.673267 20625 ts_tablet_manager.cc:1905] T 068238cff6cf447cb19adfe4cd35c47c P 3f0b35a9cc30436591f8fcf99f08c91e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:29:02.673267 20627 ts_tablet_manager.cc:1905] T 068238cff6cf447cb19adfe4cd35c47c P b0734d5da4c5446cb5087ba3554fefbe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:29:02.675838 20625 ts_tablet_manager.cc:1918] T 068238cff6cf447cb19adfe4cd35c47c P 3f0b35a9cc30436591f8fcf99f08c91e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:29:02.675860 20627 ts_tablet_manager.cc:1918] T 068238cff6cf447cb19adfe4cd35c47c P b0734d5da4c5446cb5087ba3554fefbe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:29:02.675935 20627 log.cc:1196] T 068238cff6cf447cb19adfe4cd35c47c P b0734d5da4c5446cb5087ba3554fefbe: Deleting WAL directory at /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929496330398-392-0/minicluster-data/ts-1/wal/wals/068238cff6cf447cb19adfe4cd35c47c
I20260620 04:29:02.675943 20625 log.cc:1196] T 068238cff6cf447cb19adfe4cd35c47c P 3f0b35a9cc30436591f8fcf99f08c91e: Deleting WAL directory at /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929496330398-392-0/minicluster-data/ts-2/wal/wals/068238cff6cf447cb19adfe4cd35c47c
I20260620 04:29:02.676250 20625 ts_tablet_manager.cc:1939] T 068238cff6cf447cb19adfe4cd35c47c P 3f0b35a9cc30436591f8fcf99f08c91e: Deleting consensus metadata
I20260620 04:29:02.676250 20627 ts_tablet_manager.cc:1939] T 068238cff6cf447cb19adfe4cd35c47c P b0734d5da4c5446cb5087ba3554fefbe: Deleting consensus metadata
I20260620 04:29:02.677331 20120 catalog_manager.cc:4841] TS b0734d5da4c5446cb5087ba3554fefbe (127.0.98.2:35551): tablet 068238cff6cf447cb19adfe4cd35c47c (table test-table [id=a19b8f580ac34407b8dbbe055972f7c5]) successfully deleted
I20260620 04:29:02.677506 20120 catalog_manager.cc:4841] TS 3f0b35a9cc30436591f8fcf99f08c91e (127.0.98.3:39963): tablet 068238cff6cf447cb19adfe4cd35c47c (table test-table [id=a19b8f580ac34407b8dbbe055972f7c5]) successfully deleted
W20260620 04:29:02.680929 20122 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38135 (error 108)
I20260620 04:29:02.681089 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 20326
I20260620 04:29:02.682830 20597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:02.682566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38135 (local address 127.0.0.1:34298)
0620 04:29:02.682697 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 04:29:02.682759 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 04:29:02.682974 20122 catalog_manager.cc:4584] TS 8f858b533d01485694303c8788d6d2a7 (127.0.98.1:38135): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 068238cff6cf447cb19adfe4cd35c47c: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38135: connect: Connection refused (error 111)
I20260620 04:29:02.685781 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 20457
I20260620 04:29:02.690117 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 20104
2026-06-20T04:29:02Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1176 ms)
[----------] 6 tests from EncryptionPolicyTest (10031 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781929496330398-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:29:02 dist-test-slave-nk6d krb5kdc[20631](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:29:02 dist-test-slave-nk6d krb5kdc[20631](info): set up 2 sockets
Jun 20 04:29:02 dist-test-slave-nk6d krb5kdc[20631](info): commencing operation
krb5kdc: starting...
W20260620 04:29:03.647781 392 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781929496330398-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781929496330398-392-0/krb5kdc/test-user.keytab.
Jun 20 04:29:03 dist-test-slave-nk6d krb5kdc[20631](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929743, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:29:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:29:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781929496330398-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781929496330398-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 04:29:03.755985 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781929496330398-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781929496330398-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781929496330398-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41063
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781929496330398-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41063
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:29:03.860409 20644 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:29:03.860657 20644 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:29:03.860713 20644 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:29:03.864607 20644 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:29:03.864656 20644 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:29:03.864681 20644 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:29:03.864703 20644 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:29:03.864730 20644 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:29:03.869789 20644 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781929496330398-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781929496330398-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41063
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781929496330398-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41063
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20644
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781929496330398-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:29:03.870788 20644 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:29:03.871729 20644 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:29:03.879558 20649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:29:03.879570 20652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:29:03.879699 20644 server_base.cc:1034] running on GCE node
W20260620 04:29:03.879566 20650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:29:03.880358 20644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:29:03.881348 20644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:29:03.882520 20644 hybrid_clock.cc:648] HybridClock initialized: now 1781929743882498 us; error 31 us; skew 500 ppm
Jun 20 04:29:03 dist-test-slave-nk6d krb5kdc[20631](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929743, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:29:03.885164 20644 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 04:29:03.885960 20644 webserver.cc:460] Webserver started at http://127.0.0.1:38289/ using document root <none> and password file <none>
I20260620 04:29:03.886452 20644 fs_manager.cc:359] Metadata directory not provided
I20260620 04:29:03.886509 20644 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:29:03.886677 20644 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:29:03.888608 20644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781929496330398-392-0/minicluster-data/master-0/data/instance:
uuid: "3f95e20283ec4dc29b10847156b14ac5"
format_stamp: "Formatted at 2026-06-20 04:29:03 on dist-test-slave-nk6d"
server_key: "65527d44ac97f4427fc5c26c5bfc9b06"
server_key_iv: "6409a280e090dcf98045f6c0c3190080"
server_key_version: "encryptionkey@0"
I20260620 04:29:03.889092 20644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781929496330398-392-0/minicluster-data/master-0/wal/instance:
uuid: "3f95e20283ec4dc29b10847156b14ac5"
format_stamp: "Formatted at 2026-06-20 04:29:03 on dist-test-slave-nk6d"
server_key: "65527d44ac97f4427fc5c26c5bfc9b06"
server_key_iv: "6409a280e090dcf98045f6c0c3190080"
server_key_version: "encryptionkey@0"
I20260620 04:29:03.892426 20644 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 04:29:03.894686 20659 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:29:03.895907 20644 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:29:03.896057 20644 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781929496330398-392-0/minicluster-data/master-0/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781929496330398-392-0/minicluster-data/master-0/wal
uuid: "3f95e20283ec4dc29b10847156b14ac5"
format_stamp: "Formatted at 2026-06-20 04:29:03 on dist-test-slave-nk6d"
server_key: "65527d44ac97f4427fc5c26c5bfc9b06"
server_key_iv: "6409a280e090dcf98045f6c0c3190080"
server_key_version: "encryptionkey@0"
I20260620 04:29:03.896148 20644 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781929496330398-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781929496330398-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781929496330398-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:29:03.912106 20644 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:29:03.914855 20644 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:29:03.914999 20644 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:29:03.921450 20644 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41063
I20260620 04:29:03.921494 20711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41063 every 8 connection(s)
I20260620 04:29:03.922348 20644 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:29:03.925052 20712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:29:03.930368 20712 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f95e20283ec4dc29b10847156b14ac5: Bootstrap starting.
I20260620 04:29:03.931080 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 20644
I20260620 04:29:03.931208 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781929496330398-392-0/minicluster-data/master-0/wal/instance
I20260620 04:29:03.931478 392 external_mini_cluster.cc:1428] Setting key 4f78576e86bdde6855efe84671d6b12c
I20260620 04:29:03.933010 20712 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3f95e20283ec4dc29b10847156b14ac5: Neither blocks nor log segments found. Creating new log.
I20260620 04:29:03.933722 20712 log.cc:824] T 00000000000000000000000000000000 P 3f95e20283ec4dc29b10847156b14ac5: Log is configured to *not* fsync() on all Append() calls
I20260620 04:29:03.935680 20712 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f95e20283ec4dc29b10847156b14ac5: No bootstrap required, opened a new log
I20260620 04:29:03.939785 20712 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f95e20283ec4dc29b10847156b14ac5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f95e20283ec4dc29b10847156b14ac5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41063 } }
I20260620 04:29:03.940088 20712 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3f95e20283ec4dc29b10847156b14ac5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:29:03.940155 20712 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f95e20283ec4dc29b10847156b14ac5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f95e20283ec4dc29b10847156b14ac5, State: Initialized, Role: FOLLOWER
Jun 20 04:29:03 dist-test-slave-nk6d krb5kdc[20631](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929743, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 04:29:03.940554 20712 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f95e20283ec4dc29b10847156b14ac5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f95e20283ec4dc29b10847156b14ac5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41063 } }
I20260620 04:29:03.940699 20712 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3f95e20283ec4dc29b10847156b14ac5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:29:03.940760 20712 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f95e20283ec4dc29b10847156b14ac5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:29:03.940850 20712 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f95e20283ec4dc29b10847156b14ac5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:29:03.941875 20712 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f95e20283ec4dc29b10847156b14ac5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f95e20283ec4dc29b10847156b14ac5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41063 } }
I20260620 04:29:03.942251 20712 leader_election.cc:302] T 00000000000000000000000000000000 P 3f95e20283ec4dc29b10847156b14ac5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3f95e20283ec4dc29b10847156b14ac5; no voters:
I20260620 04:29:03.942607 20712 leader_election.cc:288] T 00000000000000000000000000000000 P 3f95e20283ec4dc29b10847156b14ac5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:29:03.942732 20717 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3f95e20283ec4dc29b10847156b14ac5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:29:03.942939 20717 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3f95e20283ec4dc29b10847156b14ac5 [term 1 LEADER]: Becoming Leader. State: Replica: 3f95e20283ec4dc29b10847156b14ac5, State: Running, Role: LEADER
I20260620 04:29:03.943243 20717 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f95e20283ec4dc29b10847156b14ac5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f95e20283ec4dc29b10847156b14ac5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41063 } }
I20260620 04:29:03.943881 20712 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f95e20283ec4dc29b10847156b14ac5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:29:03.944880 20719 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f95e20283ec4dc29b10847156b14ac5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f95e20283ec4dc29b10847156b14ac5. Latest consensus state: current_term: 1 leader_uuid: "3f95e20283ec4dc29b10847156b14ac5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f95e20283ec4dc29b10847156b14ac5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41063 } } }
I20260620 04:29:03.944995 20719 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f95e20283ec4dc29b10847156b14ac5 [sys.catalog]: This master's current role is: LEADER
I20260620 04:29:03.945322 20718 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f95e20283ec4dc29b10847156b14ac5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3f95e20283ec4dc29b10847156b14ac5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f95e20283ec4dc29b10847156b14ac5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41063 } } }
I20260620 04:29:03.945405 20718 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f95e20283ec4dc29b10847156b14ac5 [sys.catalog]: This master's current role is: LEADER
I20260620 04:29:03.946895 20715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:03.932649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50679 (local address 127.0.0.1:41063)
0620 04:29:03.933066 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:03.933082 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:03.933124 (+ 42us) server_negotiation.cc:407] Connection header received
0620 04:29:03.933810 (+ 686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:03.933828 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:03.934195 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:03.934486 (+ 291us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:29:03.935335 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:03.936416 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:03.937607 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:03.938094 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:03.938217 (+ 123us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:03.941228 (+ 3011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:29:03.941234 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:29:03.941251 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:29:03.941281 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:29:03.943123 (+ 1842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:29:03.943758 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:29:03.943764 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:29:03.943769 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:29:03.943907 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:29:03.944294 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:29:03.944297 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:29:03.944299 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:29:03.944817 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:29:03.945755 (+ 938us) server_negotiation.cc:1035] Waiting for connection context
0620 04:29:03.945916 (+ 161us) server_negotiation.cc:299] Negotiation successful
0620 04:29:03.946140 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":124,"threads_started":1}
W20260620 04:29:03.948448 20732 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3f95e20283ec4dc29b10847156b14ac5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 04:29:03.948514 20732 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 04:29:03.948567 20726 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:29:03.949362 20726 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:29:03.949786 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 04:29:03.954684 20726 catalog_manager.cc:1347] Generated new cluster ID: 8e06823a474545a7b2d28187a336cccf
I20260620 04:29:03.954736 20726 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:29:03.960067 20715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:03.951662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58101 (local address 127.0.0.1:41063)
0620 04:29:03.951754 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:03.951758 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:03.951994 (+ 236us) server_negotiation.cc:407] Connection header received
0620 04:29:03.952154 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:03.952157 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:03.952200 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:03.952270 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:29:03.953031 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:03.953691 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:03.954846 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:03.955292 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:03.955400 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:03.956943 (+ 1543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:29:03.956946 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:29:03.956949 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:29:03.956971 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:29:03.958463 (+ 1492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:29:03.958940 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:29:03.958943 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:29:03.958944 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:29:03.959015 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:29:03.959302 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:29:03.959304 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:29:03.959306 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:29:03.959472 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:29:03.959584 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 04:29:03.959816 (+ 232us) server_negotiation.cc:299] Negotiation successful
0620 04:29:03.959896 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 04:29:03.979627 20726 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:29:03.980893 20726 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:29:03.989938 20726 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3f95e20283ec4dc29b10847156b14ac5: Generated new TSK 0
I20260620 04:29:03.990581 20726 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 04:29:03.998788 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 20644
W20260620 04:29:04.003263 20629 connection.cc:535] client connection to 127.0.0.1:41063 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41063): Connection reset by peer (error 104)
2026-06-20T04:29:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1307 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781929496330398-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:29:04 dist-test-slave-nk6d krb5kdc[20743](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:29:04 dist-test-slave-nk6d krb5kdc[20743](info): set up 2 sockets
Jun 20 04:29:04 dist-test-slave-nk6d krb5kdc[20743](info): commencing operation
krb5kdc: starting...
W20260620 04:29:04.947796 392 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781929496330398-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781929496330398-392-0/krb5kdc/test-user.keytab.
Jun 20 04:29:05 dist-test-slave-nk6d krb5kdc[20743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929745, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:29:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:29:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781929496330398-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781929496330398-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 04:29:05.056922 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781929496330398-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781929496330398-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781929496330398-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34991
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781929496330398-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34991
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:29:05.163872 20756 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:29:05.164199 20756 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:29:05.164268 20756 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:29:05.168221 20756 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:29:05.168290 20756 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:29:05.168318 20756 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:29:05.168337 20756 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:29:05.168363 20756 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:29:05.173593 20756 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781929496330398-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781929496330398-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34991
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781929496330398-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34991
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20756
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781929496330398-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:29:05.174654 20756 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:29:05.175657 20756 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:29:05.183945 20762 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:29:05.184111 20756 server_base.cc:1034] running on GCE node
W20260620 04:29:05.183934 20764 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:29:05.184060 20761 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:29:05.184880 20756 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:29:05.185999 20756 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:29:05.187196 20756 hybrid_clock.cc:648] HybridClock initialized: now 1781929745187166 us; error 38 us; skew 500 ppm
Jun 20 04:29:05 dist-test-slave-nk6d krb5kdc[20743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929745, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:29:05.189834 20756 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 04:29:05.190647 20756 webserver.cc:460] Webserver started at http://127.0.0.1:39869/ using document root <none> and password file <none>
I20260620 04:29:05.191154 20756 fs_manager.cc:359] Metadata directory not provided
I20260620 04:29:05.191208 20756 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:29:05.191372 20756 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:29:05.193336 20756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781929496330398-392-0/minicluster-data/master-0/data/instance:
uuid: "7ccbf3afed62414ba58f58654751d30e"
format_stamp: "Formatted at 2026-06-20 04:29:05 on dist-test-slave-nk6d"
server_key: "82fbb7cb3d18c6ce61a7b09ce5e01902"
server_key_iv: "25bcba03e451d22f582787760cb1f4eb"
server_key_version: "encryptionkey@0"
I20260620 04:29:05.193851 20756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781929496330398-392-0/minicluster-data/master-0/wal/instance:
uuid: "7ccbf3afed62414ba58f58654751d30e"
format_stamp: "Formatted at 2026-06-20 04:29:05 on dist-test-slave-nk6d"
server_key: "82fbb7cb3d18c6ce61a7b09ce5e01902"
server_key_iv: "25bcba03e451d22f582787760cb1f4eb"
server_key_version: "encryptionkey@0"
I20260620 04:29:05.197320 20756 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 04:29:05.199510 20771 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:29:05.200758 20756 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:29:05.200875 20756 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781929496330398-392-0/minicluster-data/master-0/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781929496330398-392-0/minicluster-data/master-0/wal
uuid: "7ccbf3afed62414ba58f58654751d30e"
format_stamp: "Formatted at 2026-06-20 04:29:05 on dist-test-slave-nk6d"
server_key: "82fbb7cb3d18c6ce61a7b09ce5e01902"
server_key_iv: "25bcba03e451d22f582787760cb1f4eb"
server_key_version: "encryptionkey@0"
I20260620 04:29:05.200975 20756 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781929496330398-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781929496330398-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781929496330398-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:29:05.223177 20756 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:29:05.225936 20756 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:29:05.226082 20756 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:29:05.232416 20756 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34991
I20260620 04:29:05.232445 20823 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34991 every 8 connection(s)
I20260620 04:29:05.233321 20756 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:29:05.236017 20824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:29:05.241233 20824 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ccbf3afed62414ba58f58654751d30e: Bootstrap starting.
I20260620 04:29:05.242668 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 20756
I20260620 04:29:05.242790 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781929496330398-392-0/minicluster-data/master-0/wal/instance
I20260620 04:29:05.243041 392 external_mini_cluster.cc:1428] Setting key a8d19de11732ece44b8d9ab6cfca3328
I20260620 04:29:05.243614 20824 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7ccbf3afed62414ba58f58654751d30e: Neither blocks nor log segments found. Creating new log.
I20260620 04:29:05.244477 20824 log.cc:824] T 00000000000000000000000000000000 P 7ccbf3afed62414ba58f58654751d30e: Log is configured to *not* fsync() on all Append() calls
I20260620 04:29:05.246165 20824 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ccbf3afed62414ba58f58654751d30e: No bootstrap required, opened a new log
I20260620 04:29:05.249473 20824 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7ccbf3afed62414ba58f58654751d30e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ccbf3afed62414ba58f58654751d30e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34991 } }
I20260620 04:29:05.249678 20824 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7ccbf3afed62414ba58f58654751d30e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:29:05.249742 20824 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7ccbf3afed62414ba58f58654751d30e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ccbf3afed62414ba58f58654751d30e, State: Initialized, Role: FOLLOWER
I20260620 04:29:05.250136 20824 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ccbf3afed62414ba58f58654751d30e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ccbf3afed62414ba58f58654751d30e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34991 } }
I20260620 04:29:05.250257 20824 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7ccbf3afed62414ba58f58654751d30e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:29:05.250329 20824 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7ccbf3afed62414ba58f58654751d30e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:29:05.250427 20824 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7ccbf3afed62414ba58f58654751d30e [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:29:05.251439 20824 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7ccbf3afed62414ba58f58654751d30e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ccbf3afed62414ba58f58654751d30e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34991 } }
I20260620 04:29:05.251497 20827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:05.244080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51791 (local address 127.0.0.1:34991)
0620 04:29:05.244501 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:05.244515 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:05.244550 (+ 35us) server_negotiation.cc:407] Connection header received
0620 04:29:05.245219 (+ 669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:05.245236 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:05.245557 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:05.245871 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:29:05.246241 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:05.247225 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:05.248099 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:05.248547 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:05.248697 (+ 150us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:05.249568 (+ 871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:29:05.249573 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:29:05.249585 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:29:05.250078 (+ 493us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:29:05.250133 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:29:05.250156 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:29:05.250271 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 04:29:05.250454 (+ 183us) server_negotiation.cc:299] Negotiation successful
0620 04:29:05.250585 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":128,"threads_started":1}
I20260620 04:29:05.251776 20824 leader_election.cc:302] T 00000000000000000000000000000000 P 7ccbf3afed62414ba58f58654751d30e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7ccbf3afed62414ba58f58654751d30e; no voters:
I20260620 04:29:05.252067 20824 leader_election.cc:288] T 00000000000000000000000000000000 P 7ccbf3afed62414ba58f58654751d30e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:29:05.252147 20829 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7ccbf3afed62414ba58f58654751d30e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:29:05.252302 20829 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7ccbf3afed62414ba58f58654751d30e [term 1 LEADER]: Becoming Leader. State: Replica: 7ccbf3afed62414ba58f58654751d30e, State: Running, Role: LEADER
I20260620 04:29:05.252584 20829 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7ccbf3afed62414ba58f58654751d30e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ccbf3afed62414ba58f58654751d30e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34991 } }
I20260620 04:29:05.254058 20824 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7ccbf3afed62414ba58f58654751d30e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:29:05.254233 20831 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ccbf3afed62414ba58f58654751d30e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ccbf3afed62414ba58f58654751d30e. Latest consensus state: current_term: 1 leader_uuid: "7ccbf3afed62414ba58f58654751d30e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ccbf3afed62414ba58f58654751d30e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34991 } } }
I20260620 04:29:05.254413 20831 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ccbf3afed62414ba58f58654751d30e [sys.catalog]: This master's current role is: LEADER
I20260620 04:29:05.254277 20830 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ccbf3afed62414ba58f58654751d30e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7ccbf3afed62414ba58f58654751d30e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ccbf3afed62414ba58f58654751d30e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34991 } } }
I20260620 04:29:05.254519 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 04:29:05.254523 20830 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ccbf3afed62414ba58f58654751d30e [sys.catalog]: This master's current role is: LEADER
I20260620 04:29:05.254873 20833 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:29:05.258005 20833 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:29:05.262176 20827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:05.257581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40745 (local address 127.0.0.1:34991)
0620 04:29:05.257678 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:05.257682 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:05.258494 (+ 812us) server_negotiation.cc:407] Connection header received
0620 04:29:05.258719 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:05.258722 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:05.258765 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:05.258846 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:29:05.259123 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:05.259861 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:05.260681 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:05.260928 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:05.261012 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:05.261752 (+ 740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:29:05.261756 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:29:05.261758 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:29:05.261778 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:29:05.261803 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:29:05.261813 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:29:05.261876 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 04:29:05.262017 (+ 141us) server_negotiation.cc:299] Negotiation successful
0620 04:29:05.262047 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 04:29:05.263543 20833 catalog_manager.cc:1347] Generated new cluster ID: 7341c3cbbe4d4e6c9efcc4497a7b5161
I20260620 04:29:05.263605 20833 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:29:05.279820 20833 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:29:05.280789 20833 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:29:05.288744 20833 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7ccbf3afed62414ba58f58654751d30e: Generated new TSK 0
I20260620 04:29:05.289417 20833 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 04:29:05.296815 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 20756
W20260620 04:29:05.301681 20741 connection.cc:535] client connection to 127.0.0.1:34991 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34991): Connection reset by peer (error 104)
2026-06-20T04:29:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1298 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781929496330398-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:29:05 dist-test-slave-nk6d krb5kdc[20855](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:29:05 dist-test-slave-nk6d krb5kdc[20855](info): set up 2 sockets
Jun 20 04:29:05 dist-test-slave-nk6d krb5kdc[20855](info): commencing operation
krb5kdc: starting...
W20260620 04:29:06.239830 392 mini_kdc.cc:121] Time spent starting KDC: real 0.924s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781929496330398-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781929496330398-392-0/krb5kdc/test-user.keytab.
Jun 20 04:29:06 dist-test-slave-nk6d krb5kdc[20855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929746, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:29:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:29:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781929496330398-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781929496330398-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 04:29:06.348269 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781929496330398-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781929496330398-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781929496330398-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43201
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781929496330398-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43201
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:29:06.455973 20868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:29:06.456243 20868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:29:06.456283 20868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:29:06.460225 20868 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:29:06.460276 20868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:29:06.460294 20868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:29:06.460310 20868 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:29:06.460337 20868 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:29:06.465485 20868 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781929496330398-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781929496330398-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43201
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781929496330398-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43201
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20868
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781929496330398-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:29:06.466454 20868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:29:06.467417 20868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:29:06.475298 20873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:29:06.475346 20876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:29:06.475490 20868 server_base.cc:1034] running on GCE node
W20260620 04:29:06.475495 20874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:29:06.476171 20868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:29:06.477198 20868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:29:06.478354 20868 hybrid_clock.cc:648] HybridClock initialized: now 1781929746478342 us; error 33 us; skew 500 ppm
Jun 20 04:29:06 dist-test-slave-nk6d krb5kdc[20855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929746, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:29:06.480928 20868 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 04:29:06.481703 20868 webserver.cc:460] Webserver started at http://127.0.0.1:36843/ using document root <none> and password file <none>
I20260620 04:29:06.482180 20868 fs_manager.cc:359] Metadata directory not provided
I20260620 04:29:06.482234 20868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:29:06.482398 20868 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:29:06.484331 20868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781929496330398-392-0/minicluster-data/master-0/data/instance:
uuid: "f8aae9a09f2f4c019d4c915ff939c856"
format_stamp: "Formatted at 2026-06-20 04:29:06 on dist-test-slave-nk6d"
server_key: "a4c16517cf2ec226404ba3eaaae57f2f"
server_key_iv: "9bb6ef6fbf3765e0edf352ae266604ce"
server_key_version: "encryptionkey@0"
I20260620 04:29:06.484817 20868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781929496330398-392-0/minicluster-data/master-0/wal/instance:
uuid: "f8aae9a09f2f4c019d4c915ff939c856"
format_stamp: "Formatted at 2026-06-20 04:29:06 on dist-test-slave-nk6d"
server_key: "a4c16517cf2ec226404ba3eaaae57f2f"
server_key_iv: "9bb6ef6fbf3765e0edf352ae266604ce"
server_key_version: "encryptionkey@0"
I20260620 04:29:06.488235 20868 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 04:29:06.490460 20883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:29:06.491583 20868 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:29:06.491674 20868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781929496330398-392-0/minicluster-data/master-0/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781929496330398-392-0/minicluster-data/master-0/wal
uuid: "f8aae9a09f2f4c019d4c915ff939c856"
format_stamp: "Formatted at 2026-06-20 04:29:06 on dist-test-slave-nk6d"
server_key: "a4c16517cf2ec226404ba3eaaae57f2f"
server_key_iv: "9bb6ef6fbf3765e0edf352ae266604ce"
server_key_version: "encryptionkey@0"
I20260620 04:29:06.491750 20868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781929496330398-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781929496330398-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781929496330398-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:29:06.495900 20868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:29:06.498792 20868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:29:06.498931 20868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:29:06.505156 20868 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43201
I20260620 04:29:06.505194 20935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43201 every 8 connection(s)
I20260620 04:29:06.506022 20868 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:29:06.508661 20936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:29:06.513475 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 20868
I20260620 04:29:06.513615 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781929496330398-392-0/minicluster-data/master-0/wal/instance
I20260620 04:29:06.513883 392 external_mini_cluster.cc:1428] Setting key 8eeb4f3de504e80c6a6189c080cf5505
I20260620 04:29:06.513937 20936 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8aae9a09f2f4c019d4c915ff939c856: Bootstrap starting.
I20260620 04:29:06.516296 20936 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f8aae9a09f2f4c019d4c915ff939c856: Neither blocks nor log segments found. Creating new log.
I20260620 04:29:06.517004 20936 log.cc:824] T 00000000000000000000000000000000 P f8aae9a09f2f4c019d4c915ff939c856: Log is configured to *not* fsync() on all Append() calls
I20260620 04:29:06.518723 20936 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8aae9a09f2f4c019d4c915ff939c856: No bootstrap required, opened a new log
I20260620 04:29:06.519160 20939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:06.515181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43535 (local address 127.0.0.1:43201)
0620 04:29:06.515632 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:06.515647 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:06.515683 (+ 36us) server_negotiation.cc:407] Connection header received
0620 04:29:06.516433 (+ 750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:06.516456 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:06.516743 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:06.517067 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:29:06.517969 (+ 902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:29:06.517973 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:29:06.517983 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:29:06.517991 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:29:06.518036 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:29:06.518055 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:29:06.518155 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 04:29:06.518303 (+ 148us) server_negotiation.cc:299] Negotiation successful
0620 04:29:06.518456 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":121,"threads_started":1}
I20260620 04:29:06.521858 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 04:29:06.522538 20936 raft_consensus.cc:348] T 00000000000000000000000000000000 P f8aae9a09f2f4c019d4c915ff939c856 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8aae9a09f2f4c019d4c915ff939c856" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43201 } }
I20260620 04:29:06.522739 20936 raft_consensus.cc:374] T 00000000000000000000000000000000 P f8aae9a09f2f4c019d4c915ff939c856 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:29:06.522806 20936 raft_consensus.cc:729] T 00000000000000000000000000000000 P f8aae9a09f2f4c019d4c915ff939c856 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8aae9a09f2f4c019d4c915ff939c856, State: Initialized, Role: FOLLOWER
I20260620 04:29:06.523234 20936 consensus_queue.cc:260] T 00000000000000000000000000000000 P f8aae9a09f2f4c019d4c915ff939c856 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8aae9a09f2f4c019d4c915ff939c856" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43201 } }
I20260620 04:29:06.523355 20936 raft_consensus.cc:388] T 00000000000000000000000000000000 P f8aae9a09f2f4c019d4c915ff939c856 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:29:06.523412 20936 raft_consensus.cc:482] T 00000000000000000000000000000000 P f8aae9a09f2f4c019d4c915ff939c856 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:29:06.523504 20936 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f8aae9a09f2f4c019d4c915ff939c856 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:29:06.524436 20936 raft_consensus.cc:504] T 00000000000000000000000000000000 P f8aae9a09f2f4c019d4c915ff939c856 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8aae9a09f2f4c019d4c915ff939c856" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43201 } }
I20260620 04:29:06.524734 20936 leader_election.cc:302] T 00000000000000000000000000000000 P f8aae9a09f2f4c019d4c915ff939c856 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f8aae9a09f2f4c019d4c915ff939c856; no voters:
I20260620 04:29:06.524999 20936 leader_election.cc:288] T 00000000000000000000000000000000 P f8aae9a09f2f4c019d4c915ff939c856 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:29:06.525071 20946 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f8aae9a09f2f4c019d4c915ff939c856 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:29:06.525276 20946 raft_consensus.cc:686] T 00000000000000000000000000000000 P f8aae9a09f2f4c019d4c915ff939c856 [term 1 LEADER]: Becoming Leader. State: Replica: f8aae9a09f2f4c019d4c915ff939c856, State: Running, Role: LEADER
I20260620 04:29:06.525287 20939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:06.523706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48335 (local address 127.0.0.1:43201)
0620 04:29:06.523804 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:06.523808 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:06.523995 (+ 187us) server_negotiation.cc:407] Connection header received
0620 04:29:06.524103 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:06.524106 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:06.524137 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:06.524198 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:29:06.524841 (+ 643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:29:06.524845 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:29:06.524848 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:29:06.524858 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:29:06.524884 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:29:06.524896 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:29:06.524965 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 04:29:06.525027 (+ 62us) server_negotiation.cc:299] Negotiation successful
0620 04:29:06.525060 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 04:29:06.525823 20946 consensus_queue.cc:237] T 00000000000000000000000000000000 P f8aae9a09f2f4c019d4c915ff939c856 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8aae9a09f2f4c019d4c915ff939c856" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43201 } }
I20260620 04:29:06.526252 20936 sys_catalog.cc:564] T 00000000000000000000000000000000 P f8aae9a09f2f4c019d4c915ff939c856 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:29:06.527544 20948 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8aae9a09f2f4c019d4c915ff939c856 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f8aae9a09f2f4c019d4c915ff939c856. Latest consensus state: current_term: 1 leader_uuid: "f8aae9a09f2f4c019d4c915ff939c856" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8aae9a09f2f4c019d4c915ff939c856" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43201 } } }
I20260620 04:29:06.527570 20947 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8aae9a09f2f4c019d4c915ff939c856 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f8aae9a09f2f4c019d4c915ff939c856" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8aae9a09f2f4c019d4c915ff939c856" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43201 } } }
I20260620 04:29:06.527652 20948 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8aae9a09f2f4c019d4c915ff939c856 [sys.catalog]: This master's current role is: LEADER
I20260620 04:29:06.527662 20947 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8aae9a09f2f4c019d4c915ff939c856 [sys.catalog]: This master's current role is: LEADER
W20260620 04:29:06.530848 20961 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f8aae9a09f2f4c019d4c915ff939c856: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 04:29:06.530910 20961 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 04:29:06.530983 20956 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:29:06.531651 20956 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:29:06.537062 20956 catalog_manager.cc:1347] Generated new cluster ID: 8a8f22dd49114564a68609511ede0874
I20260620 04:29:06.537117 20956 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:29:06.557657 20956 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:29:06.558050 20956 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:29:06.564829 20956 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f8aae9a09f2f4c019d4c915ff939c856: Generated new TSK 0
I20260620 04:29:06.565503 20956 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 04:29:06.567909 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 20868
2026-06-20T04:29:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1271 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781929496330398-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:29:06 dist-test-slave-nk6d krb5kdc[20967](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:29:06 dist-test-slave-nk6d krb5kdc[20967](info): set up 2 sockets
Jun 20 04:29:06 dist-test-slave-nk6d krb5kdc[20967](info): commencing operation
krb5kdc: starting...
W20260620 04:29:07.511790 392 mini_kdc.cc:121] Time spent starting KDC: real 0.925s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781929496330398-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781929496330398-392-0/krb5kdc/test-user.keytab.
Jun 20 04:29:07 dist-test-slave-nk6d krb5kdc[20967](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929747, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:29:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:29:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781929496330398-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781929496330398-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 04:29:07.627519 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781929496330398-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781929496330398-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781929496330398-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39849
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781929496330398-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39849
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:29:07.735442 20980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:29:07.735702 20980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:29:07.735757 20980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:29:07.739796 20980 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:29:07.739864 20980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:29:07.739892 20980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:29:07.739912 20980 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:29:07.739969 20980 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:29:07.745261 20980 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781929496330398-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781929496330398-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39849
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781929496330398-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39849
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20980
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781929496330398-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:29:07.746282 20980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:29:07.747296 20980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:29:07.755275 20988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:29:07.755373 20986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:29:07.755422 20985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:29:07.755687 20980 server_base.cc:1034] running on GCE node
I20260620 04:29:07.756316 20980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:29:07.757407 20980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:29:07.758589 20980 hybrid_clock.cc:648] HybridClock initialized: now 1781929747758559 us; error 40 us; skew 500 ppm
Jun 20 04:29:07 dist-test-slave-nk6d krb5kdc[20967](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929747, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:29:07.765535 20980 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 04:29:07.766417 20980 webserver.cc:460] Webserver started at http://127.0.0.1:40923/ using document root <none> and password file <none>
I20260620 04:29:07.766942 20980 fs_manager.cc:359] Metadata directory not provided
I20260620 04:29:07.767005 20980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:29:07.767186 20980 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:29:07.769176 20980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781929496330398-392-0/minicluster-data/master-0/data/instance:
uuid: "f01c60ccedde45ca993dd0e1abb04488"
format_stamp: "Formatted at 2026-06-20 04:29:07 on dist-test-slave-nk6d"
server_key: "1f1fed19dca191801ff7ddc3990433c2"
server_key_iv: "47ea304843b9844a4df13f312141b57f"
server_key_version: "encryptionkey@0"
I20260620 04:29:07.769691 20980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781929496330398-392-0/minicluster-data/master-0/wal/instance:
uuid: "f01c60ccedde45ca993dd0e1abb04488"
format_stamp: "Formatted at 2026-06-20 04:29:07 on dist-test-slave-nk6d"
server_key: "1f1fed19dca191801ff7ddc3990433c2"
server_key_iv: "47ea304843b9844a4df13f312141b57f"
server_key_version: "encryptionkey@0"
I20260620 04:29:07.773224 20980 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 04:29:07.775653 20995 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:29:07.776933 20980 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:29:07.777047 20980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781929496330398-392-0/minicluster-data/master-0/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781929496330398-392-0/minicluster-data/master-0/wal
uuid: "f01c60ccedde45ca993dd0e1abb04488"
format_stamp: "Formatted at 2026-06-20 04:29:07 on dist-test-slave-nk6d"
server_key: "1f1fed19dca191801ff7ddc3990433c2"
server_key_iv: "47ea304843b9844a4df13f312141b57f"
server_key_version: "encryptionkey@0"
I20260620 04:29:07.777168 20980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781929496330398-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781929496330398-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781929496330398-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:29:07.801769 20980 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:29:07.804975 20980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:29:07.805181 20980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:29:07.812000 20980 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39849
I20260620 04:29:07.812047 21047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39849 every 8 connection(s)
I20260620 04:29:07.812939 20980 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:29:07.813597 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 20980
I20260620 04:29:07.813740 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781929496330398-392-0/minicluster-data/master-0/wal/instance
I20260620 04:29:07.814036 392 external_mini_cluster.cc:1428] Setting key 3535c733f68bbbaa35ddf7e9b32e19e8
I20260620 04:29:07.816169 21048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:29:07.822464 21048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f01c60ccedde45ca993dd0e1abb04488: Bootstrap starting.
Jun 20 04:29:07 dist-test-slave-nk6d krb5kdc[20967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929747, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 04:29:07.825024 21048 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f01c60ccedde45ca993dd0e1abb04488: Neither blocks nor log segments found. Creating new log.
I20260620 04:29:07.825829 21048 log.cc:824] T 00000000000000000000000000000000 P f01c60ccedde45ca993dd0e1abb04488: Log is configured to *not* fsync() on all Append() calls
I20260620 04:29:07.827893 21048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f01c60ccedde45ca993dd0e1abb04488: No bootstrap required, opened a new log
I20260620 04:29:07.828418 21051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:07.815367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51531 (local address 127.0.0.1:39849)
0620 04:29:07.815766 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:07.815781 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:07.815817 (+ 36us) server_negotiation.cc:407] Connection header received
0620 04:29:07.816501 (+ 684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:07.816518 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:07.816861 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:07.817141 (+ 280us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:29:07.817976 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:07.818951 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:07.820120 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:07.820590 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:07.820712 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:07.823456 (+ 2744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:29:07.823461 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:29:07.823474 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:29:07.823501 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:29:07.825373 (+ 1872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:29:07.825855 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:29:07.825860 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:29:07.825864 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:29:07.826069 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:29:07.826360 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:29:07.826363 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:29:07.826365 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:29:07.826836 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:29:07.827014 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0620 04:29:07.827315 (+ 301us) server_negotiation.cc:299] Negotiation successful
0620 04:29:07.827590 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":104,"threads_started":1}
I20260620 04:29:07.831334 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 04:29:07.831736 21048 raft_consensus.cc:348] T 00000000000000000000000000000000 P f01c60ccedde45ca993dd0e1abb04488 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f01c60ccedde45ca993dd0e1abb04488" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39849 } }
I20260620 04:29:07.832023 21048 raft_consensus.cc:374] T 00000000000000000000000000000000 P f01c60ccedde45ca993dd0e1abb04488 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:29:07.832082 21048 raft_consensus.cc:729] T 00000000000000000000000000000000 P f01c60ccedde45ca993dd0e1abb04488 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f01c60ccedde45ca993dd0e1abb04488, State: Initialized, Role: FOLLOWER
I20260620 04:29:07.832549 21048 consensus_queue.cc:260] T 00000000000000000000000000000000 P f01c60ccedde45ca993dd0e1abb04488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f01c60ccedde45ca993dd0e1abb04488" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39849 } }
I20260620 04:29:07.832718 21048 raft_consensus.cc:388] T 00000000000000000000000000000000 P f01c60ccedde45ca993dd0e1abb04488 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:29:07.832787 21048 raft_consensus.cc:482] T 00000000000000000000000000000000 P f01c60ccedde45ca993dd0e1abb04488 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:29:07.832880 21048 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f01c60ccedde45ca993dd0e1abb04488 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:29:07.834024 21048 raft_consensus.cc:504] T 00000000000000000000000000000000 P f01c60ccedde45ca993dd0e1abb04488 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f01c60ccedde45ca993dd0e1abb04488" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39849 } }
I20260620 04:29:07.834385 21048 leader_election.cc:302] T 00000000000000000000000000000000 P f01c60ccedde45ca993dd0e1abb04488 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f01c60ccedde45ca993dd0e1abb04488; no voters:
I20260620 04:29:07.834714 21048 leader_election.cc:288] T 00000000000000000000000000000000 P f01c60ccedde45ca993dd0e1abb04488 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:29:07.834796 21058 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f01c60ccedde45ca993dd0e1abb04488 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:29:07.835000 21058 raft_consensus.cc:686] T 00000000000000000000000000000000 P f01c60ccedde45ca993dd0e1abb04488 [term 1 LEADER]: Becoming Leader. State: Replica: f01c60ccedde45ca993dd0e1abb04488, State: Running, Role: LEADER
I20260620 04:29:07.835374 21058 consensus_queue.cc:237] T 00000000000000000000000000000000 P f01c60ccedde45ca993dd0e1abb04488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f01c60ccedde45ca993dd0e1abb04488" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39849 } }
I20260620 04:29:07.835810 21048 sys_catalog.cc:564] T 00000000000000000000000000000000 P f01c60ccedde45ca993dd0e1abb04488 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:29:07.836941 21059 sys_catalog.cc:455] T 00000000000000000000000000000000 P f01c60ccedde45ca993dd0e1abb04488 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f01c60ccedde45ca993dd0e1abb04488" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f01c60ccedde45ca993dd0e1abb04488" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39849 } } }
I20260620 04:29:07.837069 21059 sys_catalog.cc:458] T 00000000000000000000000000000000 P f01c60ccedde45ca993dd0e1abb04488 [sys.catalog]: This master's current role is: LEADER
I20260620 04:29:07.837445 21060 sys_catalog.cc:455] T 00000000000000000000000000000000 P f01c60ccedde45ca993dd0e1abb04488 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f01c60ccedde45ca993dd0e1abb04488. Latest consensus state: current_term: 1 leader_uuid: "f01c60ccedde45ca993dd0e1abb04488" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f01c60ccedde45ca993dd0e1abb04488" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39849 } } }
I20260620 04:29:07.837538 21060 sys_catalog.cc:458] T 00000000000000000000000000000000 P f01c60ccedde45ca993dd0e1abb04488 [sys.catalog]: This master's current role is: LEADER
W20260620 04:29:07.840459 21073 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f01c60ccedde45ca993dd0e1abb04488: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 04:29:07.840543 21073 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 04:29:07.840596 21068 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:29:07.841392 21068 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:29:07.841917 21051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:07.833477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55993 (local address 127.0.0.1:39849)
0620 04:29:07.833575 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:07.833592 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:07.833794 (+ 202us) server_negotiation.cc:407] Connection header received
0620 04:29:07.833912 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:07.833914 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:07.833947 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:07.834005 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:29:07.834761 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:07.835478 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:07.836629 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:07.836987 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:07.837076 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:07.838700 (+ 1624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:29:07.838704 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:29:07.838706 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:29:07.838729 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:29:07.840211 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:29:07.840708 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:29:07.840712 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:29:07.840713 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:29:07.840795 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:29:07.841089 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:29:07.841092 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:29:07.841094 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:29:07.841304 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:29:07.841429 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 04:29:07.841663 (+ 234us) server_negotiation.cc:299] Negotiation successful
0620 04:29:07.841779 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 04:29:07.847013 21068 catalog_manager.cc:1347] Generated new cluster ID: d895222dcf7448b3833bcc56d20090a4
I20260620 04:29:07.847110 21068 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:29:07.866300 21068 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:29:07.867233 21068 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:29:07.876003 21068 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f01c60ccedde45ca993dd0e1abb04488: Generated new TSK 0
I20260620 04:29:07.876662 21068 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 04:29:07.883692 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 20980
2026-06-20T04:29:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1315 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781929496330398-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:29:07 dist-test-slave-nk6d krb5kdc[21079](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:29:07 dist-test-slave-nk6d krb5kdc[21079](info): set up 2 sockets
Jun 20 04:29:07 dist-test-slave-nk6d krb5kdc[21079](info): commencing operation
krb5kdc: starting...
W20260620 04:29:08.835886 392 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781929496330398-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781929496330398-392-0/krb5kdc/test-user.keytab.
Jun 20 04:29:08 dist-test-slave-nk6d krb5kdc[21079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:29:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:29:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781929496330398-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781929496330398-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 04:29:08.947011 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781929496330398-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781929496330398-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781929496330398-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36867
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781929496330398-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36867
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:29:09.052363 21092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:29:09.052623 21092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:29:09.052670 21092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:29:09.056699 21092 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:29:09.056759 21092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:29:09.056778 21092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:29:09.056793 21092 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:29:09.056807 21092 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:29:09.061973 21092 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781929496330398-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781929496330398-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36867
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781929496330398-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36867
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21092
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781929496330398-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:29:09.062981 21092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:29:09.063944 21092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:29:09.071847 21098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:29:09.071846 21097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:29:09.071851 21100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:29:09.072192 21092 server_base.cc:1034] running on GCE node
I20260620 04:29:09.072714 21092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:29:09.073769 21092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:29:09.074937 21092 hybrid_clock.cc:648] HybridClock initialized: now 1781929749074921 us; error 32 us; skew 500 ppm
Jun 20 04:29:09 dist-test-slave-nk6d krb5kdc[21079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929749, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:29:09.077574 21092 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 04:29:09.078433 21092 webserver.cc:460] Webserver started at http://127.0.0.1:34717/ using document root <none> and password file <none>
I20260620 04:29:09.078955 21092 fs_manager.cc:359] Metadata directory not provided
I20260620 04:29:09.079017 21092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:29:09.079210 21092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:29:09.081211 21092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781929496330398-392-0/minicluster-data/master-0/data/instance:
uuid: "d4d6a4d2e1f14bc48ffe731e631a0ae5"
format_stamp: "Formatted at 2026-06-20 04:29:09 on dist-test-slave-nk6d"
server_key: "570d34cab1104b750ca85fa1cbfa5b26"
server_key_iv: "905075c3720dbfe8457b41c097f5f39e"
server_key_version: "encryptionkey@0"
I20260620 04:29:09.081735 21092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781929496330398-392-0/minicluster-data/master-0/wal/instance:
uuid: "d4d6a4d2e1f14bc48ffe731e631a0ae5"
format_stamp: "Formatted at 2026-06-20 04:29:09 on dist-test-slave-nk6d"
server_key: "570d34cab1104b750ca85fa1cbfa5b26"
server_key_iv: "905075c3720dbfe8457b41c097f5f39e"
server_key_version: "encryptionkey@0"
I20260620 04:29:09.085347 21092 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 04:29:09.087756 21107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:29:09.088958 21092 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:29:09.089068 21092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781929496330398-392-0/minicluster-data/master-0/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781929496330398-392-0/minicluster-data/master-0/wal
uuid: "d4d6a4d2e1f14bc48ffe731e631a0ae5"
format_stamp: "Formatted at 2026-06-20 04:29:09 on dist-test-slave-nk6d"
server_key: "570d34cab1104b750ca85fa1cbfa5b26"
server_key_iv: "905075c3720dbfe8457b41c097f5f39e"
server_key_version: "encryptionkey@0"
I20260620 04:29:09.089164 21092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781929496330398-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781929496330398-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781929496330398-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:29:09.113708 21092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:29:09.116892 21092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:29:09.117079 21092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:29:09.123883 21092 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36867
I20260620 04:29:09.123904 21159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36867 every 8 connection(s)
I20260620 04:29:09.124835 21092 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:29:09.127568 21160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:29:09.132794 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 21092
I20260620 04:29:09.132881 21160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4d6a4d2e1f14bc48ffe731e631a0ae5: Bootstrap starting.
I20260620 04:29:09.132923 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781929496330398-392-0/minicluster-data/master-0/wal/instance
I20260620 04:29:09.133162 392 external_mini_cluster.cc:1428] Setting key 7d271ee09b3a615f2682758be1d0710c
I20260620 04:29:09.135286 21160 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d4d6a4d2e1f14bc48ffe731e631a0ae5: Neither blocks nor log segments found. Creating new log.
I20260620 04:29:09.136003 21160 log.cc:824] T 00000000000000000000000000000000 P d4d6a4d2e1f14bc48ffe731e631a0ae5: Log is configured to *not* fsync() on all Append() calls
I20260620 04:29:09.137720 21160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4d6a4d2e1f14bc48ffe731e631a0ae5: No bootstrap required, opened a new log
I20260620 04:29:09.141086 21160 raft_consensus.cc:348] T 00000000000000000000000000000000 P d4d6a4d2e1f14bc48ffe731e631a0ae5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4d6a4d2e1f14bc48ffe731e631a0ae5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36867 } }
I20260620 04:29:09.141307 21160 raft_consensus.cc:374] T 00000000000000000000000000000000 P d4d6a4d2e1f14bc48ffe731e631a0ae5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:29:09.141361 21160 raft_consensus.cc:729] T 00000000000000000000000000000000 P d4d6a4d2e1f14bc48ffe731e631a0ae5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4d6a4d2e1f14bc48ffe731e631a0ae5, State: Initialized, Role: FOLLOWER
I20260620 04:29:09.141415 21163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:09.134486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40511 (local address 127.0.0.1:36867)
0620 04:29:09.134866 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:09.134881 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:09.134914 (+ 33us) server_negotiation.cc:407] Connection header received
0620 04:29:09.135561 (+ 647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:09.135580 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:09.135925 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:09.136286 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:29:09.136664 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:09.137597 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:09.138461 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:09.138919 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:09.139058 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:09.139773 (+ 715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:29:09.139779 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:29:09.139790 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:29:09.140203 (+ 413us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:29:09.140262 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:29:09.140286 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:29:09.140367 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 04:29:09.140498 (+ 131us) server_negotiation.cc:299] Negotiation successful
0620 04:29:09.140685 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":111,"threads_started":1}
I20260620 04:29:09.141819 21160 consensus_queue.cc:260] T 00000000000000000000000000000000 P d4d6a4d2e1f14bc48ffe731e631a0ae5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4d6a4d2e1f14bc48ffe731e631a0ae5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36867 } }
I20260620 04:29:09.141973 21160 raft_consensus.cc:388] T 00000000000000000000000000000000 P d4d6a4d2e1f14bc48ffe731e631a0ae5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:29:09.142031 21160 raft_consensus.cc:482] T 00000000000000000000000000000000 P d4d6a4d2e1f14bc48ffe731e631a0ae5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:29:09.142117 21160 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d4d6a4d2e1f14bc48ffe731e631a0ae5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:29:09.142997 21160 raft_consensus.cc:504] T 00000000000000000000000000000000 P d4d6a4d2e1f14bc48ffe731e631a0ae5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4d6a4d2e1f14bc48ffe731e631a0ae5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36867 } }
I20260620 04:29:09.143357 21160 leader_election.cc:302] T 00000000000000000000000000000000 P d4d6a4d2e1f14bc48ffe731e631a0ae5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d4d6a4d2e1f14bc48ffe731e631a0ae5; no voters:
I20260620 04:29:09.143662 21160 leader_election.cc:288] T 00000000000000000000000000000000 P d4d6a4d2e1f14bc48ffe731e631a0ae5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:29:09.143786 21165 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d4d6a4d2e1f14bc48ffe731e631a0ae5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:29:09.144021 21165 raft_consensus.cc:686] T 00000000000000000000000000000000 P d4d6a4d2e1f14bc48ffe731e631a0ae5 [term 1 LEADER]: Becoming Leader. State: Replica: d4d6a4d2e1f14bc48ffe731e631a0ae5, State: Running, Role: LEADER
I20260620 04:29:09.144302 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 04:29:09.144387 21165 consensus_queue.cc:237] T 00000000000000000000000000000000 P d4d6a4d2e1f14bc48ffe731e631a0ae5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4d6a4d2e1f14bc48ffe731e631a0ae5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36867 } }
I20260620 04:29:09.144925 21160 sys_catalog.cc:564] T 00000000000000000000000000000000 P d4d6a4d2e1f14bc48ffe731e631a0ae5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:29:09.146041 21166 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4d6a4d2e1f14bc48ffe731e631a0ae5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d4d6a4d2e1f14bc48ffe731e631a0ae5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4d6a4d2e1f14bc48ffe731e631a0ae5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36867 } } }
I20260620 04:29:09.146226 21166 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4d6a4d2e1f14bc48ffe731e631a0ae5 [sys.catalog]: This master's current role is: LEADER
I20260620 04:29:09.146528 21166 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4d6a4d2e1f14bc48ffe731e631a0ae5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4d6a4d2e1f14bc48ffe731e631a0ae5. Latest consensus state: current_term: 1 leader_uuid: "d4d6a4d2e1f14bc48ffe731e631a0ae5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4d6a4d2e1f14bc48ffe731e631a0ae5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36867 } } }
I20260620 04:29:09.146659 21166 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4d6a4d2e1f14bc48ffe731e631a0ae5 [sys.catalog]: This master's current role is: LEADER
W20260620 04:29:09.149873 21184 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d4d6a4d2e1f14bc48ffe731e631a0ae5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 04:29:09.149946 21184 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 04:29:09.150240 21181 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:29:09.151031 21181 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:29:09.151458 21163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:09.146562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59801 (local address 127.0.0.1:36867)
0620 04:29:09.146668 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:09.146673 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:09.147305 (+ 632us) server_negotiation.cc:407] Connection header received
0620 04:29:09.147479 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:09.147482 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:09.147539 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:09.147588 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:29:09.147924 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:09.148676 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:09.149764 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:09.150077 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:09.150158 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:09.151042 (+ 884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:29:09.151046 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:29:09.151049 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:29:09.151061 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:29:09.151090 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:29:09.151102 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:29:09.151174 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 04:29:09.151278 (+ 104us) server_negotiation.cc:299] Negotiation successful
0620 04:29:09.151331 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 04:29:09.156541 21181 catalog_manager.cc:1347] Generated new cluster ID: b09406c70ada43fb9ec4a73c3e509a4c
I20260620 04:29:09.156598 21181 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:29:09.162827 21181 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:29:09.164026 21181 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:29:09.180537 21181 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d4d6a4d2e1f14bc48ffe731e631a0ae5: Generated new TSK 0
I20260620 04:29:09.181427 21181 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 04:29:09.185740 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 21092
2026-06-20T04:29:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1302 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781929496330398-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:29:09 dist-test-slave-nk6d krb5kdc[21191](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:29:09 dist-test-slave-nk6d krb5kdc[21191](info): set up 2 sockets
Jun 20 04:29:09 dist-test-slave-nk6d krb5kdc[21191](info): commencing operation
krb5kdc: starting...
W20260620 04:29:10.119889 392 mini_kdc.cc:121] Time spent starting KDC: real 0.915s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781929496330398-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781929496330398-392-0/krb5kdc/test-user.keytab.
Jun 20 04:29:10 dist-test-slave-nk6d krb5kdc[21191](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929750, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:29:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:29:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781929496330398-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781929496330398-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 04:29:10.228839 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781929496330398-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781929496330398-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781929496330398-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44867
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781929496330398-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44867
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:29:10.333464 21204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:29:10.333719 21204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:29:10.333760 21204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:29:10.337682 21204 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:29:10.337742 21204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:29:10.337760 21204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:29:10.337776 21204 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:29:10.337795 21204 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:29:10.343000 21204 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781929496330398-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781929496330398-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44867
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781929496330398-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44867
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21204
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781929496330398-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:29:10.344045 21204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:29:10.345008 21204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:29:10.353055 21209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:29:10.353179 21204 server_base.cc:1034] running on GCE node
W20260620 04:29:10.353030 21210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:29:10.353040 21212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:29:10.353891 21204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:29:10.354974 21204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:29:10.356127 21204 hybrid_clock.cc:648] HybridClock initialized: now 1781929750356107 us; error 28 us; skew 500 ppm
Jun 20 04:29:10 dist-test-slave-nk6d krb5kdc[21191](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929750, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:29:10.358803 21204 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 04:29:10.359634 21204 webserver.cc:460] Webserver started at http://127.0.0.1:44599/ using document root <none> and password file <none>
I20260620 04:29:10.360185 21204 fs_manager.cc:359] Metadata directory not provided
I20260620 04:29:10.360250 21204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:29:10.360459 21204 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:29:10.362416 21204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781929496330398-392-0/minicluster-data/master-0/data/instance:
uuid: "a4c342554ce94c55804ac75bfb166379"
format_stamp: "Formatted at 2026-06-20 04:29:10 on dist-test-slave-nk6d"
server_key: "7a5015f9176de1ab5495c7e08daede4f"
server_key_iv: "41a9c2146a2d4b84fc96bc77115e4419"
server_key_version: "encryptionkey@0"
I20260620 04:29:10.362941 21204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781929496330398-392-0/minicluster-data/master-0/wal/instance:
uuid: "a4c342554ce94c55804ac75bfb166379"
format_stamp: "Formatted at 2026-06-20 04:29:10 on dist-test-slave-nk6d"
server_key: "7a5015f9176de1ab5495c7e08daede4f"
server_key_iv: "41a9c2146a2d4b84fc96bc77115e4419"
server_key_version: "encryptionkey@0"
I20260620 04:29:10.366429 21204 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 04:29:10.368748 21219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:29:10.369946 21204 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:29:10.370054 21204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781929496330398-392-0/minicluster-data/master-0/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781929496330398-392-0/minicluster-data/master-0/wal
uuid: "a4c342554ce94c55804ac75bfb166379"
format_stamp: "Formatted at 2026-06-20 04:29:10 on dist-test-slave-nk6d"
server_key: "7a5015f9176de1ab5495c7e08daede4f"
server_key_iv: "41a9c2146a2d4b84fc96bc77115e4419"
server_key_version: "encryptionkey@0"
I20260620 04:29:10.370146 21204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781929496330398-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781929496330398-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781929496330398-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:29:10.374527 21204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:29:10.377345 21204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:29:10.377502 21204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:29:10.384142 21204 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44867
I20260620 04:29:10.384183 21271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44867 every 8 connection(s)
I20260620 04:29:10.385015 21204 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:29:10.387610 21272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:29:10.392944 21272 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4c342554ce94c55804ac75bfb166379: Bootstrap starting.
I20260620 04:29:10.394124 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 21204
I20260620 04:29:10.394265 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781929496330398-392-0/minicluster-data/master-0/wal/instance
I20260620 04:29:10.394551 392 external_mini_cluster.cc:1428] Setting key 507a3fd33d47cb817ebfedcaa784f465
I20260620 04:29:10.395742 21272 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a4c342554ce94c55804ac75bfb166379: Neither blocks nor log segments found. Creating new log.
I20260620 04:29:10.396519 21272 log.cc:824] T 00000000000000000000000000000000 P a4c342554ce94c55804ac75bfb166379: Log is configured to *not* fsync() on all Append() calls
I20260620 04:29:10.398340 21272 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4c342554ce94c55804ac75bfb166379: No bootstrap required, opened a new log
I20260620 04:29:10.399271 21275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:10.395615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51897 (local address 127.0.0.1:44867)
0620 04:29:10.396087 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:10.396100 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:10.396136 (+ 36us) server_negotiation.cc:407] Connection header received
0620 04:29:10.396826 (+ 690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:10.396845 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:10.397122 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:10.397441 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:29:10.398148 (+ 707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:29:10.398153 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:29:10.398164 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:29:10.398173 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:29:10.398223 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:29:10.398241 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:29:10.398334 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 04:29:10.398443 (+ 109us) server_negotiation.cc:299] Negotiation successful
0620 04:29:10.398597 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":138,"threads_started":1}
I20260620 04:29:10.401803 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 04:29:10.402132 21272 raft_consensus.cc:348] T 00000000000000000000000000000000 P a4c342554ce94c55804ac75bfb166379 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4c342554ce94c55804ac75bfb166379" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44867 } }
I20260620 04:29:10.402349 21272 raft_consensus.cc:374] T 00000000000000000000000000000000 P a4c342554ce94c55804ac75bfb166379 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:29:10.402413 21272 raft_consensus.cc:729] T 00000000000000000000000000000000 P a4c342554ce94c55804ac75bfb166379 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4c342554ce94c55804ac75bfb166379, State: Initialized, Role: FOLLOWER
I20260620 04:29:10.402818 21272 consensus_queue.cc:260] T 00000000000000000000000000000000 P a4c342554ce94c55804ac75bfb166379 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4c342554ce94c55804ac75bfb166379" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44867 } }
I20260620 04:29:10.403045 21272 raft_consensus.cc:388] T 00000000000000000000000000000000 P a4c342554ce94c55804ac75bfb166379 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:29:10.403156 21272 raft_consensus.cc:482] T 00000000000000000000000000000000 P a4c342554ce94c55804ac75bfb166379 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:29:10.403255 21272 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a4c342554ce94c55804ac75bfb166379 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:29:10.404354 21272 raft_consensus.cc:504] T 00000000000000000000000000000000 P a4c342554ce94c55804ac75bfb166379 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4c342554ce94c55804ac75bfb166379" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44867 } }
I20260620 04:29:10.404742 21272 leader_election.cc:302] T 00000000000000000000000000000000 P a4c342554ce94c55804ac75bfb166379 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a4c342554ce94c55804ac75bfb166379; no voters:
I20260620 04:29:10.405123 21272 leader_election.cc:288] T 00000000000000000000000000000000 P a4c342554ce94c55804ac75bfb166379 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:29:10.405197 21282 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a4c342554ce94c55804ac75bfb166379 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:29:10.405413 21282 raft_consensus.cc:686] T 00000000000000000000000000000000 P a4c342554ce94c55804ac75bfb166379 [term 1 LEADER]: Becoming Leader. State: Replica: a4c342554ce94c55804ac75bfb166379, State: Running, Role: LEADER
I20260620 04:29:10.405445 21275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:10.403694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53531 (local address 127.0.0.1:44867)
0620 04:29:10.403835 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:10.403840 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:10.404121 (+ 281us) server_negotiation.cc:407] Connection header received
0620 04:29:10.404245 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:10.404247 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:10.404277 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:10.404337 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:29:10.405019 (+ 682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:29:10.405021 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:29:10.405023 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:29:10.405033 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:29:10.405057 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:29:10.405067 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:29:10.405140 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 04:29:10.405258 (+ 118us) server_negotiation.cc:299] Negotiation successful
0620 04:29:10.405294 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 04:29:10.405753 21282 consensus_queue.cc:237] T 00000000000000000000000000000000 P a4c342554ce94c55804ac75bfb166379 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4c342554ce94c55804ac75bfb166379" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44867 } }
I20260620 04:29:10.406425 21272 sys_catalog.cc:564] T 00000000000000000000000000000000 P a4c342554ce94c55804ac75bfb166379 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:29:10.406948 21282 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4c342554ce94c55804ac75bfb166379 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a4c342554ce94c55804ac75bfb166379. Latest consensus state: current_term: 1 leader_uuid: "a4c342554ce94c55804ac75bfb166379" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4c342554ce94c55804ac75bfb166379" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44867 } } }
I20260620 04:29:10.407119 21282 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4c342554ce94c55804ac75bfb166379 [sys.catalog]: This master's current role is: LEADER
I20260620 04:29:10.407372 21283 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4c342554ce94c55804ac75bfb166379 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a4c342554ce94c55804ac75bfb166379" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4c342554ce94c55804ac75bfb166379" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44867 } } }
I20260620 04:29:10.407507 21283 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4c342554ce94c55804ac75bfb166379 [sys.catalog]: This master's current role is: LEADER
I20260620 04:29:10.407581 21289 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:29:10.410789 21289 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:29:10.415932 21289 catalog_manager.cc:1347] Generated new cluster ID: 577408bcabb046e8b3c78435aa51dd73
I20260620 04:29:10.416045 21289 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:29:10.430262 21289 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:29:10.430559 21289 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:29:10.440075 21289 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a4c342554ce94c55804ac75bfb166379: Generated new TSK 0
I20260620 04:29:10.440763 21289 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 04:29:10.447114 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 21204
2026-06-20T04:29:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1261 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781929496330398-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:29:10 dist-test-slave-nk6d krb5kdc[21303](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:29:10 dist-test-slave-nk6d krb5kdc[21303](info): set up 2 sockets
Jun 20 04:29:10 dist-test-slave-nk6d krb5kdc[21303](info): commencing operation
krb5kdc: starting...
W20260620 04:29:11.403878 392 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781929496330398-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781929496330398-392-0/krb5kdc/test-user.keytab.
Jun 20 04:29:11 dist-test-slave-nk6d krb5kdc[21303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929751, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:29:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:29:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781929496330398-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781929496330398-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 04:29:11.513098 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781929496330398-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781929496330398-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781929496330398-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36351
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781929496330398-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36351
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:29:11.620874 21316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:29:11.621148 21316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:29:11.621191 21316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:29:11.625120 21316 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:29:11.625182 21316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:29:11.625201 21316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:29:11.625216 21316 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:29:11.625231 21316 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:29:11.630529 21316 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781929496330398-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781929496330398-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36351
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781929496330398-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36351
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21316
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781929496330398-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:29:11.631593 21316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:29:11.632587 21316 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:29:11.640616 21321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:29:11.640722 21324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:29:11.640637 21322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:29:11.640920 21316 server_base.cc:1034] running on GCE node
I20260620 04:29:11.641467 21316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:29:11.642565 21316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:29:11.643743 21316 hybrid_clock.cc:648] HybridClock initialized: now 1781929751643724 us; error 34 us; skew 500 ppm
Jun 20 04:29:11 dist-test-slave-nk6d krb5kdc[21303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929751, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:29:11.646380 21316 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 04:29:11.647281 21316 webserver.cc:460] Webserver started at http://127.0.0.1:39455/ using document root <none> and password file <none>
I20260620 04:29:11.647871 21316 fs_manager.cc:359] Metadata directory not provided
I20260620 04:29:11.647933 21316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:29:11.648172 21316 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:29:11.650193 21316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781929496330398-392-0/minicluster-data/master-0/data/instance:
uuid: "59dad0ca38c74d9483d0999b6f098730"
format_stamp: "Formatted at 2026-06-20 04:29:11 on dist-test-slave-nk6d"
server_key: "5606103be6dd0eed0d0196ab1a3ff73e"
server_key_iv: "2f7542a93fdbc6246aa509f294478967"
server_key_version: "encryptionkey@0"
I20260620 04:29:11.650703 21316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781929496330398-392-0/minicluster-data/master-0/wal/instance:
uuid: "59dad0ca38c74d9483d0999b6f098730"
format_stamp: "Formatted at 2026-06-20 04:29:11 on dist-test-slave-nk6d"
server_key: "5606103be6dd0eed0d0196ab1a3ff73e"
server_key_iv: "2f7542a93fdbc6246aa509f294478967"
server_key_version: "encryptionkey@0"
I20260620 04:29:11.654307 21316 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 04:29:11.656575 21331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:29:11.657846 21316 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:29:11.657968 21316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781929496330398-392-0/minicluster-data/master-0/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781929496330398-392-0/minicluster-data/master-0/wal
uuid: "59dad0ca38c74d9483d0999b6f098730"
format_stamp: "Formatted at 2026-06-20 04:29:11 on dist-test-slave-nk6d"
server_key: "5606103be6dd0eed0d0196ab1a3ff73e"
server_key_iv: "2f7542a93fdbc6246aa509f294478967"
server_key_version: "encryptionkey@0"
I20260620 04:29:11.658069 21316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781929496330398-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781929496330398-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781929496330398-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:29:11.673962 21316 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:29:11.676930 21316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:29:11.677107 21316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:29:11.685024 21316 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36351
I20260620 04:29:11.685004 21383 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36351 every 8 connection(s)
I20260620 04:29:11.686362 21316 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:29:11.688596 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 21316
I20260620 04:29:11.688714 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781929496330398-392-0/minicluster-data/master-0/wal/instance
I20260620 04:29:11.689006 392 external_mini_cluster.cc:1428] Setting key 7c2c3a11ccf724c7272bbc813015dd14
I20260620 04:29:11.689375 21384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:29:11.696067 21384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59dad0ca38c74d9483d0999b6f098730: Bootstrap starting.
Jun 20 04:29:11 dist-test-slave-nk6d krb5kdc[21303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929751, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 04:29:11.698665 21384 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 59dad0ca38c74d9483d0999b6f098730: Neither blocks nor log segments found. Creating new log.
I20260620 04:29:11.699386 21384 log.cc:824] T 00000000000000000000000000000000 P 59dad0ca38c74d9483d0999b6f098730: Log is configured to *not* fsync() on all Append() calls
I20260620 04:29:11.701468 21384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59dad0ca38c74d9483d0999b6f098730: No bootstrap required, opened a new log
I20260620 04:29:11.703154 21387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:11.690335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.32:37699 (local address 127.0.0.1:36351)
0620 04:29:11.690748 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:11.690764 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:11.690800 (+ 36us) server_negotiation.cc:407] Connection header received
0620 04:29:11.691440 (+ 640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:11.691493 (+ 53us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:11.691840 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:11.692161 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:29:11.692991 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:11.694075 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:11.695229 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:11.695712 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:11.695831 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:11.698600 (+ 2769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:29:11.698605 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:29:11.698618 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:29:11.698645 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:29:11.700328 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:29:11.700884 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:29:11.700889 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:29:11.700893 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:29:11.701003 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:29:11.701314 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:29:11.701316 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:29:11.701318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:29:11.701728 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:29:11.701884 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 04:29:11.702185 (+ 301us) server_negotiation.cc:299] Negotiation successful
0620 04:29:11.702392 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":123,"threads_started":1}
I20260620 04:29:11.705032 21384 raft_consensus.cc:348] T 00000000000000000000000000000000 P 59dad0ca38c74d9483d0999b6f098730 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59dad0ca38c74d9483d0999b6f098730" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36351 } }
I20260620 04:29:11.705292 21384 raft_consensus.cc:374] T 00000000000000000000000000000000 P 59dad0ca38c74d9483d0999b6f098730 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:29:11.705358 21384 raft_consensus.cc:729] T 00000000000000000000000000000000 P 59dad0ca38c74d9483d0999b6f098730 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59dad0ca38c74d9483d0999b6f098730, State: Initialized, Role: FOLLOWER
I20260620 04:29:11.705793 21384 consensus_queue.cc:260] T 00000000000000000000000000000000 P 59dad0ca38c74d9483d0999b6f098730 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59dad0ca38c74d9483d0999b6f098730" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36351 } }
I20260620 04:29:11.705935 21384 raft_consensus.cc:388] T 00000000000000000000000000000000 P 59dad0ca38c74d9483d0999b6f098730 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:29:11.706004 21384 raft_consensus.cc:482] T 00000000000000000000000000000000 P 59dad0ca38c74d9483d0999b6f098730 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:29:11.706101 21384 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 59dad0ca38c74d9483d0999b6f098730 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:29:11.706103 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 04:29:11.706941 21384 raft_consensus.cc:504] T 00000000000000000000000000000000 P 59dad0ca38c74d9483d0999b6f098730 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59dad0ca38c74d9483d0999b6f098730" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36351 } }
I20260620 04:29:11.707257 21384 leader_election.cc:302] T 00000000000000000000000000000000 P 59dad0ca38c74d9483d0999b6f098730 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 59dad0ca38c74d9483d0999b6f098730; no voters:
I20260620 04:29:11.707520 21384 leader_election.cc:288] T 00000000000000000000000000000000 P 59dad0ca38c74d9483d0999b6f098730 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:29:11.707616 21393 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 59dad0ca38c74d9483d0999b6f098730 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:29:11.707823 21393 raft_consensus.cc:686] T 00000000000000000000000000000000 P 59dad0ca38c74d9483d0999b6f098730 [term 1 LEADER]: Becoming Leader. State: Replica: 59dad0ca38c74d9483d0999b6f098730, State: Running, Role: LEADER
I20260620 04:29:11.708189 21393 consensus_queue.cc:237] T 00000000000000000000000000000000 P 59dad0ca38c74d9483d0999b6f098730 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59dad0ca38c74d9483d0999b6f098730" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36351 } }
I20260620 04:29:11.708750 21384 sys_catalog.cc:564] T 00000000000000000000000000000000 P 59dad0ca38c74d9483d0999b6f098730 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:29:11.709808 21395 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59dad0ca38c74d9483d0999b6f098730 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 59dad0ca38c74d9483d0999b6f098730. Latest consensus state: current_term: 1 leader_uuid: "59dad0ca38c74d9483d0999b6f098730" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59dad0ca38c74d9483d0999b6f098730" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36351 } } }
I20260620 04:29:11.709816 21394 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59dad0ca38c74d9483d0999b6f098730 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "59dad0ca38c74d9483d0999b6f098730" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59dad0ca38c74d9483d0999b6f098730" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36351 } } }
I20260620 04:29:11.709932 21395 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59dad0ca38c74d9483d0999b6f098730 [sys.catalog]: This master's current role is: LEADER
I20260620 04:29:11.709932 21394 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59dad0ca38c74d9483d0999b6f098730 [sys.catalog]: This master's current role is: LEADER
I20260620 04:29:11.710307 21403 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:29:11.713388 21403 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:29:11.718696 21387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:11.708277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.32:53487 (local address 127.0.0.1:36351)
0620 04:29:11.708378 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:11.708382 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:11.710715 (+ 2333us) server_negotiation.cc:407] Connection header received
0620 04:29:11.710885 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:11.710888 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:11.710936 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:11.711004 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:29:11.711719 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:11.712563 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:11.713658 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:11.713989 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:11.714084 (+ 95us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:11.715334 (+ 1250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:29:11.715337 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:29:11.715340 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:29:11.715362 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:29:11.716854 (+ 1492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:29:11.717416 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:29:11.717419 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:29:11.717421 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:29:11.717505 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:29:11.717816 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:29:11.717819 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:29:11.717821 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:29:11.718025 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:29:11.718159 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 04:29:11.718473 (+ 314us) server_negotiation.cc:299] Negotiation successful
0620 04:29:11.718557 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 04:29:11.719620 21403 catalog_manager.cc:1347] Generated new cluster ID: c9000db9315741e0ac7e2e9522120174
I20260620 04:29:11.719672 21403 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:29:11.731312 21403 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:29:11.732278 21403 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:29:11.735646 21403 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 59dad0ca38c74d9483d0999b6f098730: Generated new TSK 0
I20260620 04:29:11.736277 21403 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 04:29:11.747709 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 21316
W20260620 04:29:11.752249 21301 connection.cc:535] client connection to 127.0.0.1:36351 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36351): Connection reset by peer (error 104)
2026-06-20T04:29:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1300 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781929496330398-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:29:11 dist-test-slave-nk6d krb5kdc[21415](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:29:11 dist-test-slave-nk6d krb5kdc[21415](info): set up 2 sockets
Jun 20 04:29:11 dist-test-slave-nk6d krb5kdc[21415](info): commencing operation
krb5kdc: starting...
W20260620 04:29:12.695994 392 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781929496330398-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781929496330398-392-0/krb5kdc/test-user.keytab.
Jun 20 04:29:12 dist-test-slave-nk6d krb5kdc[21415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929752, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:29:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:29:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781929496330398-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781929496330398-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 04:29:12.806607 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781929496330398-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781929496330398-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781929496330398-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46255
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781929496330398-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46255
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:29:12.910974 21428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:29:12.911217 21428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:29:12.911273 21428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:29:12.915179 21428 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:29:12.915233 21428 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:29:12.915259 21428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:29:12.915280 21428 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:29:12.915305 21428 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:29:12.920416 21428 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781929496330398-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781929496330398-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46255
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781929496330398-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46255
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21428
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781929496330398-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:29:12.921348 21428 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:29:12.922286 21428 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 04:29:12.930030 21428 server_base.cc:1034] running on GCE node
W20260620 04:29:12.930023 21436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:29:12.930087 21434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:29:12.930023 21433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:29:12.930740 21428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:29:12.931725 21428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:29:12.932879 21428 hybrid_clock.cc:648] HybridClock initialized: now 1781929752932857 us; error 34 us; skew 500 ppm
Jun 20 04:29:12 dist-test-slave-nk6d krb5kdc[21415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929752, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:29:12.939296 21428 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 04:29:12.940183 21428 webserver.cc:460] Webserver started at http://127.0.0.1:37957/ using document root <none> and password file <none>
I20260620 04:29:12.940690 21428 fs_manager.cc:359] Metadata directory not provided
I20260620 04:29:12.940748 21428 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:29:12.940908 21428 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:29:12.942873 21428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781929496330398-392-0/minicluster-data/master-0/data/instance:
uuid: "51694541eee7407a881fdc59fa831296"
format_stamp: "Formatted at 2026-06-20 04:29:12 on dist-test-slave-nk6d"
server_key: "45927e23af5ac1c3c70fe4139e00854c"
server_key_iv: "ed693bebda4e3455819932ffdbc5cf4e"
server_key_version: "encryptionkey@0"
I20260620 04:29:12.943383 21428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781929496330398-392-0/minicluster-data/master-0/wal/instance:
uuid: "51694541eee7407a881fdc59fa831296"
format_stamp: "Formatted at 2026-06-20 04:29:12 on dist-test-slave-nk6d"
server_key: "45927e23af5ac1c3c70fe4139e00854c"
server_key_iv: "ed693bebda4e3455819932ffdbc5cf4e"
server_key_version: "encryptionkey@0"
I20260620 04:29:12.946887 21428 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 04:29:12.949218 21443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:29:12.950508 21428 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 04:29:12.950641 21428 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781929496330398-392-0/minicluster-data/master-0/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781929496330398-392-0/minicluster-data/master-0/wal
uuid: "51694541eee7407a881fdc59fa831296"
format_stamp: "Formatted at 2026-06-20 04:29:12 on dist-test-slave-nk6d"
server_key: "45927e23af5ac1c3c70fe4139e00854c"
server_key_iv: "ed693bebda4e3455819932ffdbc5cf4e"
server_key_version: "encryptionkey@0"
I20260620 04:29:12.950770 21428 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781929496330398-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781929496330398-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781929496330398-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:29:12.963199 21428 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:29:12.965823 21428 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:29:12.965978 21428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:29:12.972471 21428 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46255
I20260620 04:29:12.972482 21495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46255 every 8 connection(s)
I20260620 04:29:12.973335 21428 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:29:12.975924 21496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:29:12.981314 21496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51694541eee7407a881fdc59fa831296: Bootstrap starting.
I20260620 04:29:12.982089 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 21428
I20260620 04:29:12.982231 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781929496330398-392-0/minicluster-data/master-0/wal/instance
I20260620 04:29:12.982517 392 external_mini_cluster.cc:1428] Setting key 6fb854098570ebe9ed25ce39b42aaf66
I20260620 04:29:12.983928 21496 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 51694541eee7407a881fdc59fa831296: Neither blocks nor log segments found. Creating new log.
I20260620 04:29:12.984628 21496 log.cc:824] T 00000000000000000000000000000000 P 51694541eee7407a881fdc59fa831296: Log is configured to *not* fsync() on all Append() calls
I20260620 04:29:12.986338 21496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51694541eee7407a881fdc59fa831296: No bootstrap required, opened a new log
I20260620 04:29:12.989727 21496 raft_consensus.cc:348] T 00000000000000000000000000000000 P 51694541eee7407a881fdc59fa831296 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51694541eee7407a881fdc59fa831296" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46255 } }
I20260620 04:29:12.989936 21496 raft_consensus.cc:374] T 00000000000000000000000000000000 P 51694541eee7407a881fdc59fa831296 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:29:12.990005 21496 raft_consensus.cc:729] T 00000000000000000000000000000000 P 51694541eee7407a881fdc59fa831296 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51694541eee7407a881fdc59fa831296, State: Initialized, Role: FOLLOWER
I20260620 04:29:12.990402 21496 consensus_queue.cc:260] T 00000000000000000000000000000000 P 51694541eee7407a881fdc59fa831296 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51694541eee7407a881fdc59fa831296" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46255 } }
I20260620 04:29:12.990537 21496 raft_consensus.cc:388] T 00000000000000000000000000000000 P 51694541eee7407a881fdc59fa831296 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:29:12.990598 21496 raft_consensus.cc:482] T 00000000000000000000000000000000 P 51694541eee7407a881fdc59fa831296 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:29:12.990700 21496 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 51694541eee7407a881fdc59fa831296 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:29:12.990859 21499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:12.983529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.32:41181 (local address 127.0.0.1:46255)
0620 04:29:12.983897 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:12.983914 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:12.984123 (+ 209us) server_negotiation.cc:407] Connection header received
0620 04:29:12.984873 (+ 750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:12.984891 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:12.985213 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:12.985515 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:29:12.985891 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:12.986848 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:12.987691 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:12.988196 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:12.988337 (+ 141us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:12.989180 (+ 843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:29:12.989184 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:29:12.989196 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:29:12.989561 (+ 365us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:29:12.989613 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:29:12.989634 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:29:12.989752 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 04:29:12.989926 (+ 174us) server_negotiation.cc:299] Negotiation successful
0620 04:29:12.990058 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":86,"threads_started":1}
I20260620 04:29:12.991705 21496 raft_consensus.cc:504] T 00000000000000000000000000000000 P 51694541eee7407a881fdc59fa831296 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51694541eee7407a881fdc59fa831296" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46255 } }
I20260620 04:29:12.992069 21496 leader_election.cc:302] T 00000000000000000000000000000000 P 51694541eee7407a881fdc59fa831296 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 51694541eee7407a881fdc59fa831296; no voters:
I20260620 04:29:12.992337 21496 leader_election.cc:288] T 00000000000000000000000000000000 P 51694541eee7407a881fdc59fa831296 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:29:12.992419 21501 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 51694541eee7407a881fdc59fa831296 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:29:12.992579 21501 raft_consensus.cc:686] T 00000000000000000000000000000000 P 51694541eee7407a881fdc59fa831296 [term 1 LEADER]: Becoming Leader. State: Replica: 51694541eee7407a881fdc59fa831296, State: Running, Role: LEADER
I20260620 04:29:12.992856 21501 consensus_queue.cc:237] T 00000000000000000000000000000000 P 51694541eee7407a881fdc59fa831296 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51694541eee7407a881fdc59fa831296" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46255 } }
I20260620 04:29:12.993523 21496 sys_catalog.cc:564] T 00000000000000000000000000000000 P 51694541eee7407a881fdc59fa831296 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:29:12.993816 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 04:29:12.993978 21501 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51694541eee7407a881fdc59fa831296 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "51694541eee7407a881fdc59fa831296" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51694541eee7407a881fdc59fa831296" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46255 } } }
I20260620 04:29:12.994076 21501 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51694541eee7407a881fdc59fa831296 [sys.catalog]: This master's current role is: LEADER
I20260620 04:29:12.994263 21501 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51694541eee7407a881fdc59fa831296 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 51694541eee7407a881fdc59fa831296. Latest consensus state: current_term: 1 leader_uuid: "51694541eee7407a881fdc59fa831296" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51694541eee7407a881fdc59fa831296" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46255 } } }
I20260620 04:29:12.994361 21501 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51694541eee7407a881fdc59fa831296 [sys.catalog]: This master's current role is: LEADER
W20260620 04:29:12.998323 21521 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 51694541eee7407a881fdc59fa831296: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 04:29:12.998399 21521 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 04:29:12.998452 21505 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:29:12.999194 21505 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:29:13.000804 21499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:12.996073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.32:38395 (local address 127.0.0.1:46255)
0620 04:29:12.996171 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:12.996175 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:12.996508 (+ 333us) server_negotiation.cc:407] Connection header received
0620 04:29:12.996601 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:12.996604 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:12.996646 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:12.996720 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:29:12.997042 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:12.997845 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:12.998902 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:12.999261 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:12.999381 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:13.000292 (+ 911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:29:13.000297 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:29:13.000300 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:29:13.000314 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:29:13.000345 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:29:13.000360 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:29:13.000460 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 04:29:13.000623 (+ 163us) server_negotiation.cc:299] Negotiation successful
0620 04:29:13.000663 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 04:29:13.004494 21505 catalog_manager.cc:1347] Generated new cluster ID: cf8411cf1ad641cb8d234fb806775710
I20260620 04:29:13.004555 21505 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:29:13.015297 21505 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:29:13.016475 21505 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:29:13.021795 21505 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 51694541eee7407a881fdc59fa831296: Generated new TSK 0
I20260620 04:29:13.022477 21505 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 04:29:13.026604 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 21428
W20260620 04:29:13.031260 21413 connection.cc:535] client connection to 127.0.0.1:46255 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46255): Connection reset by peer (error 104)
2026-06-20T04:29:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1279 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781929496330398-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:29:13 dist-test-slave-nk6d krb5kdc[21527](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:29:13 dist-test-slave-nk6d krb5kdc[21527](info): set up 2 sockets
Jun 20 04:29:13 dist-test-slave-nk6d krb5kdc[21527](info): commencing operation
krb5kdc: starting...
W20260620 04:29:13.971892 392 mini_kdc.cc:121] Time spent starting KDC: real 0.926s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781929496330398-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781929496330398-392-0/krb5kdc/test-user.keytab.
Jun 20 04:29:14 dist-test-slave-nk6d krb5kdc[21527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:29:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:29:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781929496330398-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781929496330398-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 04:29:14.084968 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781929496330398-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781929496330398-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781929496330398-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43547
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781929496330398-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43547
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:29:14.191877 21540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:29:14.192173 21540 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:29:14.192216 21540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:29:14.196161 21540 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:29:14.196218 21540 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:29:14.196240 21540 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:29:14.196259 21540 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:29:14.196275 21540 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:29:14.201442 21540 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781929496330398-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781929496330398-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43547
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781929496330398-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43547
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21540
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781929496330398-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:29:14.202407 21540 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:29:14.203361 21540 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:29:14.211694 21546 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:29:14.211752 21545 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:29:14.211870 21540 server_base.cc:1034] running on GCE node
W20260620 04:29:14.211745 21548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:29:14.212492 21540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:29:14.213518 21540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:29:14.214674 21540 hybrid_clock.cc:648] HybridClock initialized: now 1781929754214656 us; error 29 us; skew 500 ppm
Jun 20 04:29:14 dist-test-slave-nk6d krb5kdc[21527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929754, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:29:14.217216 21540 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 04:29:14.218035 21540 webserver.cc:460] Webserver started at http://127.0.0.1:43117/ using document root <none> and password file <none>
I20260620 04:29:14.218550 21540 fs_manager.cc:359] Metadata directory not provided
I20260620 04:29:14.218613 21540 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:29:14.218791 21540 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:29:14.220783 21540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781929496330398-392-0/minicluster-data/master-0/data/instance:
uuid: "2a5fd41098a04a54826dcfda7b18bfd7"
format_stamp: "Formatted at 2026-06-20 04:29:14 on dist-test-slave-nk6d"
server_key: "0a7ac1721c09b4189dcb9afef3a6a4e8"
server_key_iv: "063059ffeb0bc515cf9a5a38e3e5c28b"
server_key_version: "encryptionkey@0"
I20260620 04:29:14.221315 21540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781929496330398-392-0/minicluster-data/master-0/wal/instance:
uuid: "2a5fd41098a04a54826dcfda7b18bfd7"
format_stamp: "Formatted at 2026-06-20 04:29:14 on dist-test-slave-nk6d"
server_key: "0a7ac1721c09b4189dcb9afef3a6a4e8"
server_key_iv: "063059ffeb0bc515cf9a5a38e3e5c28b"
server_key_version: "encryptionkey@0"
I20260620 04:29:14.224849 21540 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 04:29:14.227218 21555 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:29:14.228511 21540 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:29:14.228623 21540 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781929496330398-392-0/minicluster-data/master-0/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781929496330398-392-0/minicluster-data/master-0/wal
uuid: "2a5fd41098a04a54826dcfda7b18bfd7"
format_stamp: "Formatted at 2026-06-20 04:29:14 on dist-test-slave-nk6d"
server_key: "0a7ac1721c09b4189dcb9afef3a6a4e8"
server_key_iv: "063059ffeb0bc515cf9a5a38e3e5c28b"
server_key_version: "encryptionkey@0"
I20260620 04:29:14.228718 21540 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781929496330398-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781929496330398-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781929496330398-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:29:14.233029 21540 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:29:14.235914 21540 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:29:14.236197 21540 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:29:14.242893 21540 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43547
I20260620 04:29:14.242928 21607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43547 every 8 connection(s)
I20260620 04:29:14.243750 21540 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:29:14.246430 21608 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:29:14.250581 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 21540
I20260620 04:29:14.250727 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781929496330398-392-0/minicluster-data/master-0/wal/instance
I20260620 04:29:14.250993 392 external_mini_cluster.cc:1428] Setting key 2050eb5836239e32b7e1b0d4d98c8ec2
I20260620 04:29:14.251981 21608 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a5fd41098a04a54826dcfda7b18bfd7: Bootstrap starting.
I20260620 04:29:14.254544 21608 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a5fd41098a04a54826dcfda7b18bfd7: Neither blocks nor log segments found. Creating new log.
I20260620 04:29:14.255411 21608 log.cc:824] T 00000000000000000000000000000000 P 2a5fd41098a04a54826dcfda7b18bfd7: Log is configured to *not* fsync() on all Append() calls
I20260620 04:29:14.256335 21611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:14.252359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.32:37373 (local address 127.0.0.1:43547)
0620 04:29:14.252769 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:14.252787 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:14.252828 (+ 41us) server_negotiation.cc:407] Connection header received
0620 04:29:14.253651 (+ 823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:14.253674 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:14.254028 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:14.254381 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:29:14.255050 (+ 669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:29:14.255055 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:29:14.255068 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:29:14.255077 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:29:14.255130 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:29:14.255152 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:29:14.255260 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 04:29:14.255325 (+ 65us) server_negotiation.cc:299] Negotiation successful
0620 04:29:14.255492 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":101,"threads_started":1}
I20260620 04:29:14.257356 21608 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a5fd41098a04a54826dcfda7b18bfd7: No bootstrap required, opened a new log
I20260620 04:29:14.258883 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 04:29:14.261437 21608 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a5fd41098a04a54826dcfda7b18bfd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a5fd41098a04a54826dcfda7b18bfd7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43547 } }
I20260620 04:29:14.261706 21608 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2a5fd41098a04a54826dcfda7b18bfd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:29:14.261754 21608 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a5fd41098a04a54826dcfda7b18bfd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a5fd41098a04a54826dcfda7b18bfd7, State: Initialized, Role: FOLLOWER
I20260620 04:29:14.262204 21608 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a5fd41098a04a54826dcfda7b18bfd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a5fd41098a04a54826dcfda7b18bfd7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43547 } }
I20260620 04:29:14.262351 21608 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2a5fd41098a04a54826dcfda7b18bfd7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:29:14.262411 21608 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a5fd41098a04a54826dcfda7b18bfd7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:29:14.262506 21608 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a5fd41098a04a54826dcfda7b18bfd7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:29:14.262734 21611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:14.261028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.32:58151 (local address 127.0.0.1:43547)
0620 04:29:14.261140 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:14.261144 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:14.261303 (+ 159us) server_negotiation.cc:407] Connection header received
0620 04:29:14.261427 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:14.261429 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:14.261456 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:14.261516 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:29:14.262328 (+ 812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:29:14.262331 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:29:14.262335 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:29:14.262348 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:29:14.262379 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:29:14.262393 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:29:14.262493 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 04:29:14.262577 (+ 84us) server_negotiation.cc:299] Negotiation successful
0620 04:29:14.262617 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 04:29:14.263509 21608 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a5fd41098a04a54826dcfda7b18bfd7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a5fd41098a04a54826dcfda7b18bfd7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43547 } }
I20260620 04:29:14.263833 21608 leader_election.cc:302] T 00000000000000000000000000000000 P 2a5fd41098a04a54826dcfda7b18bfd7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a5fd41098a04a54826dcfda7b18bfd7; no voters:
I20260620 04:29:14.264119 21608 leader_election.cc:288] T 00000000000000000000000000000000 P 2a5fd41098a04a54826dcfda7b18bfd7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:29:14.264199 21618 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a5fd41098a04a54826dcfda7b18bfd7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:29:14.264415 21618 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2a5fd41098a04a54826dcfda7b18bfd7 [term 1 LEADER]: Becoming Leader. State: Replica: 2a5fd41098a04a54826dcfda7b18bfd7, State: Running, Role: LEADER
I20260620 04:29:14.264722 21618 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a5fd41098a04a54826dcfda7b18bfd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a5fd41098a04a54826dcfda7b18bfd7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43547 } }
I20260620 04:29:14.265223 21608 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a5fd41098a04a54826dcfda7b18bfd7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:29:14.266232 21620 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a5fd41098a04a54826dcfda7b18bfd7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a5fd41098a04a54826dcfda7b18bfd7. Latest consensus state: current_term: 1 leader_uuid: "2a5fd41098a04a54826dcfda7b18bfd7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a5fd41098a04a54826dcfda7b18bfd7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43547 } } }
I20260620 04:29:14.266343 21620 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a5fd41098a04a54826dcfda7b18bfd7 [sys.catalog]: This master's current role is: LEADER
I20260620 04:29:14.266585 21619 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a5fd41098a04a54826dcfda7b18bfd7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2a5fd41098a04a54826dcfda7b18bfd7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a5fd41098a04a54826dcfda7b18bfd7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43547 } } }
I20260620 04:29:14.266682 21619 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a5fd41098a04a54826dcfda7b18bfd7 [sys.catalog]: This master's current role is: LEADER
I20260620 04:29:14.266871 21627 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:29:14.269984 21627 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:29:14.275180 21627 catalog_manager.cc:1347] Generated new cluster ID: 8120acfb56c54c8d9f37c208cff51877
I20260620 04:29:14.275231 21627 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:29:14.283835 21627 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:29:14.284128 21627 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:29:14.289142 21627 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2a5fd41098a04a54826dcfda7b18bfd7: Generated new TSK 0
I20260620 04:29:14.289723 21627 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 04:29:14.297706 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 21540
2026-06-20T04:29:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1270 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781929496330398-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:29:14 dist-test-slave-nk6d krb5kdc[21639](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:29:14 dist-test-slave-nk6d krb5kdc[21639](info): set up 2 sockets
Jun 20 04:29:14 dist-test-slave-nk6d krb5kdc[21639](info): commencing operation
krb5kdc: starting...
W20260620 04:29:15.243835 392 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781929496330398-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781929496330398-392-0/krb5kdc/test-user.keytab.
Jun 20 04:29:15 dist-test-slave-nk6d krb5kdc[21639](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929755, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:29:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:29:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781929496330398-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781929496330398-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 04:29:15.355266 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781929496330398-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781929496330398-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781929496330398-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35471
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781929496330398-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35471
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:29:15.460716 21652 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:29:15.460960 21652 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:29:15.460999 21652 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:29:15.464882 21652 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:29:15.464936 21652 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:29:15.464953 21652 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:29:15.464969 21652 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:29:15.464982 21652 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:29:15.470064 21652 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781929496330398-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781929496330398-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35471
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781929496330398-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35471
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21652
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781929496330398-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:29:15.470991 21652 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:29:15.471918 21652 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:29:15.480101 21658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:29:15.480113 21660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:29:15.480165 21652 server_base.cc:1034] running on GCE node
W20260620 04:29:15.480113 21657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:29:15.480841 21652 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:29:15.481936 21652 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:29:15.483103 21652 hybrid_clock.cc:648] HybridClock initialized: now 1781929755483086 us; error 33 us; skew 500 ppm
Jun 20 04:29:15 dist-test-slave-nk6d krb5kdc[21639](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929755, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:29:15.485736 21652 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 04:29:15.486653 21652 webserver.cc:460] Webserver started at http://127.0.0.1:45891/ using document root <none> and password file <none>
I20260620 04:29:15.487187 21652 fs_manager.cc:359] Metadata directory not provided
I20260620 04:29:15.487251 21652 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:29:15.487443 21652 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:29:15.489467 21652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781929496330398-392-0/minicluster-data/master-0/data/instance:
uuid: "3370b654b03c48b8b77decb7e8cf6934"
format_stamp: "Formatted at 2026-06-20 04:29:15 on dist-test-slave-nk6d"
server_key: "be1842d15bf2b8fccffbf00edc775e5f"
server_key_iv: "2d949e84ea04a70401d73674daafdffa"
server_key_version: "encryptionkey@0"
I20260620 04:29:15.490001 21652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781929496330398-392-0/minicluster-data/master-0/wal/instance:
uuid: "3370b654b03c48b8b77decb7e8cf6934"
format_stamp: "Formatted at 2026-06-20 04:29:15 on dist-test-slave-nk6d"
server_key: "be1842d15bf2b8fccffbf00edc775e5f"
server_key_iv: "2d949e84ea04a70401d73674daafdffa"
server_key_version: "encryptionkey@0"
I20260620 04:29:15.493497 21652 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 04:29:15.495719 21667 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:29:15.496932 21652 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:29:15.497081 21652 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781929496330398-392-0/minicluster-data/master-0/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781929496330398-392-0/minicluster-data/master-0/wal
uuid: "3370b654b03c48b8b77decb7e8cf6934"
format_stamp: "Formatted at 2026-06-20 04:29:15 on dist-test-slave-nk6d"
server_key: "be1842d15bf2b8fccffbf00edc775e5f"
server_key_iv: "2d949e84ea04a70401d73674daafdffa"
server_key_version: "encryptionkey@0"
I20260620 04:29:15.497174 21652 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781929496330398-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781929496330398-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781929496330398-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:29:15.511901 21652 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:29:15.514930 21652 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:29:15.515158 21652 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:29:15.521888 21652 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35471
I20260620 04:29:15.521912 21719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35471 every 8 connection(s)
I20260620 04:29:15.522799 21652 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:29:15.525480 21720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:29:15.530642 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 21652
I20260620 04:29:15.530782 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781929496330398-392-0/minicluster-data/master-0/wal/instance
I20260620 04:29:15.530886 21720 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3370b654b03c48b8b77decb7e8cf6934: Bootstrap starting.
I20260620 04:29:15.531075 392 external_mini_cluster.cc:1428] Setting key 943268fb71d892d6e5d1da24f65d7475
I20260620 04:29:15.533391 21720 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3370b654b03c48b8b77decb7e8cf6934: Neither blocks nor log segments found. Creating new log.
I20260620 04:29:15.534094 21720 log.cc:824] T 00000000000000000000000000000000 P 3370b654b03c48b8b77decb7e8cf6934: Log is configured to *not* fsync() on all Append() calls
I20260620 04:29:15.535817 21720 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3370b654b03c48b8b77decb7e8cf6934: No bootstrap required, opened a new log
I20260620 04:29:15.539311 21720 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3370b654b03c48b8b77decb7e8cf6934 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3370b654b03c48b8b77decb7e8cf6934" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35471 } }
I20260620 04:29:15.539587 21720 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3370b654b03c48b8b77decb7e8cf6934 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:29:15.539645 21720 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3370b654b03c48b8b77decb7e8cf6934 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3370b654b03c48b8b77decb7e8cf6934, State: Initialized, Role: FOLLOWER
Jun 20 04:29:15 dist-test-slave-nk6d krb5kdc[21639](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929755, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 04:29:15.540144 21720 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3370b654b03c48b8b77decb7e8cf6934 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3370b654b03c48b8b77decb7e8cf6934" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35471 } }
I20260620 04:29:15.540274 21720 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3370b654b03c48b8b77decb7e8cf6934 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:29:15.540334 21720 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3370b654b03c48b8b77decb7e8cf6934 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:29:15.540431 21720 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3370b654b03c48b8b77decb7e8cf6934 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:29:15.541389 21720 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3370b654b03c48b8b77decb7e8cf6934 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3370b654b03c48b8b77decb7e8cf6934" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35471 } }
I20260620 04:29:15.541765 21720 leader_election.cc:302] T 00000000000000000000000000000000 P 3370b654b03c48b8b77decb7e8cf6934 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3370b654b03c48b8b77decb7e8cf6934; no voters:
I20260620 04:29:15.542066 21720 leader_election.cc:288] T 00000000000000000000000000000000 P 3370b654b03c48b8b77decb7e8cf6934 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:29:15.542142 21725 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3370b654b03c48b8b77decb7e8cf6934 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:29:15.542366 21725 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3370b654b03c48b8b77decb7e8cf6934 [term 1 LEADER]: Becoming Leader. State: Replica: 3370b654b03c48b8b77decb7e8cf6934, State: Running, Role: LEADER
I20260620 04:29:15.542718 21725 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3370b654b03c48b8b77decb7e8cf6934 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3370b654b03c48b8b77decb7e8cf6934" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35471 } }
I20260620 04:29:15.543895 21720 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3370b654b03c48b8b77decb7e8cf6934 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:29:15.543933 21725 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3370b654b03c48b8b77decb7e8cf6934 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3370b654b03c48b8b77decb7e8cf6934. Latest consensus state: current_term: 1 leader_uuid: "3370b654b03c48b8b77decb7e8cf6934" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3370b654b03c48b8b77decb7e8cf6934" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35471 } } }
I20260620 04:29:15.544152 21725 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3370b654b03c48b8b77decb7e8cf6934 [sys.catalog]: This master's current role is: LEADER
I20260620 04:29:15.544101 21727 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3370b654b03c48b8b77decb7e8cf6934 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3370b654b03c48b8b77decb7e8cf6934" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3370b654b03c48b8b77decb7e8cf6934" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35471 } } }
I20260620 04:29:15.544351 21727 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3370b654b03c48b8b77decb7e8cf6934 [sys.catalog]: This master's current role is: LEADER
I20260620 04:29:15.544579 21729 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:29:15.547493 21723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:15.532538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.32:51685 (local address 127.0.0.1:35471)
0620 04:29:15.533007 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:15.533024 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:15.533062 (+ 38us) server_negotiation.cc:407] Connection header received
0620 04:29:15.533729 (+ 667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:15.533748 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:15.534106 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:15.534439 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:29:15.535237 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:15.536293 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:15.537120 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:15.537588 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:15.537731 (+ 143us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:15.540486 (+ 2755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:29:15.540492 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:29:15.540506 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:29:15.540538 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:29:15.542645 (+ 2107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:29:15.543308 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:29:15.543314 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:29:15.543321 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:29:15.543474 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:29:15.543874 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:29:15.543877 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:29:15.543879 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:29:15.545241 (+ 1362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:29:15.546184 (+ 943us) server_negotiation.cc:1035] Waiting for connection context
0620 04:29:15.546346 (+ 162us) server_negotiation.cc:299] Negotiation successful
0620 04:29:15.546568 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":129,"threads_started":1}
I20260620 04:29:15.548975 21729 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:29:15.559324 21729 catalog_manager.cc:1347] Generated new cluster ID: da10d598d11b476bb4eb542c99e97108
I20260620 04:29:15.559401 21729 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:29:15.560242 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 04:29:15.570609 21723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:15.562337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.32:47485 (local address 127.0.0.1:35471)
0620 04:29:15.562440 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:15.562444 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:15.562644 (+ 200us) server_negotiation.cc:407] Connection header received
0620 04:29:15.562739 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:15.562742 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:15.562790 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:15.562873 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:29:15.563559 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:15.564389 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:15.565471 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:15.565839 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:15.565926 (+ 87us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:15.567419 (+ 1493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:29:15.567423 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:29:15.567426 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:29:15.567452 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:29:15.568986 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:29:15.569451 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:29:15.569455 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:29:15.569457 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:29:15.569536 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:29:15.569801 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:29:15.569804 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:29:15.569806 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:29:15.570013 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:29:15.570146 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 04:29:15.570355 (+ 209us) server_negotiation.cc:299] Negotiation successful
0620 04:29:15.570451 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 04:29:15.592214 21729 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:29:15.593207 21729 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:29:15.599872 21729 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3370b654b03c48b8b77decb7e8cf6934: Generated new TSK 0
I20260620 04:29:15.600725 21729 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 04:29:15.614248 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 21652
W20260620 04:29:15.620906 21637 connection.cc:535] client connection to 127.0.0.1:35471 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35471): Connection reset by peer (error 104)
2026-06-20T04:29:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1318 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781929496330398-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:29:15 dist-test-slave-nk6d krb5kdc[21751](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:29:15 dist-test-slave-nk6d krb5kdc[21751](info): set up 2 sockets
Jun 20 04:29:15 dist-test-slave-nk6d krb5kdc[21751](info): commencing operation
krb5kdc: starting...
W20260620 04:29:16.567864 392 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781929496330398-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781929496330398-392-0/krb5kdc/test-user.keytab.
Jun 20 04:29:16 dist-test-slave-nk6d krb5kdc[21751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929756, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:29:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:29:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781929496330398-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781929496330398-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 04:29:16.680388 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781929496330398-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781929496330398-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781929496330398-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42181
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781929496330398-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42181
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:29:16.785743 21764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:29:16.786003 21764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:29:16.786042 21764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:29:16.790038 21764 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:29:16.790103 21764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:29:16.790122 21764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:29:16.790136 21764 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:29:16.790150 21764 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:29:16.795450 21764 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781929496330398-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781929496330398-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42181
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781929496330398-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42181
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21764
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781929496330398-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:29:16.796530 21764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:29:16.797526 21764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:29:16.805529 21770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:29:16.805541 21769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:29:16.805536 21772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:29:16.805744 21764 server_base.cc:1034] running on GCE node
I20260620 04:29:16.806384 21764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:29:16.807421 21764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:29:16.808580 21764 hybrid_clock.cc:648] HybridClock initialized: now 1781929756808565 us; error 29 us; skew 500 ppm
Jun 20 04:29:16 dist-test-slave-nk6d krb5kdc[21751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929756, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:29:16.811152 21764 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 04:29:16.811998 21764 webserver.cc:460] Webserver started at http://127.0.0.1:43907/ using document root <none> and password file <none>
I20260620 04:29:16.812501 21764 fs_manager.cc:359] Metadata directory not provided
I20260620 04:29:16.812557 21764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:29:16.812717 21764 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:29:16.814640 21764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781929496330398-392-0/minicluster-data/master-0/data/instance:
uuid: "d7405de2f7254845a879225225a4e743"
format_stamp: "Formatted at 2026-06-20 04:29:16 on dist-test-slave-nk6d"
server_key: "8d19880a75e919d2de3e42b67bb483d7"
server_key_iv: "b91ee546c95790fe84c3116ad5ce7f29"
server_key_version: "encryptionkey@0"
I20260620 04:29:16.815182 21764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781929496330398-392-0/minicluster-data/master-0/wal/instance:
uuid: "d7405de2f7254845a879225225a4e743"
format_stamp: "Formatted at 2026-06-20 04:29:16 on dist-test-slave-nk6d"
server_key: "8d19880a75e919d2de3e42b67bb483d7"
server_key_iv: "b91ee546c95790fe84c3116ad5ce7f29"
server_key_version: "encryptionkey@0"
I20260620 04:29:16.818797 21764 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 04:29:16.821197 21779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:29:16.822371 21764 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 04:29:16.822467 21764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781929496330398-392-0/minicluster-data/master-0/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781929496330398-392-0/minicluster-data/master-0/wal
uuid: "d7405de2f7254845a879225225a4e743"
format_stamp: "Formatted at 2026-06-20 04:29:16 on dist-test-slave-nk6d"
server_key: "8d19880a75e919d2de3e42b67bb483d7"
server_key_iv: "b91ee546c95790fe84c3116ad5ce7f29"
server_key_version: "encryptionkey@0"
I20260620 04:29:16.822549 21764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781929496330398-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781929496330398-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781929496330398-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:29:16.841027 21764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:29:16.843729 21764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:29:16.843886 21764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:29:16.850301 21764 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42181
I20260620 04:29:16.850343 21831 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42181 every 8 connection(s)
I20260620 04:29:16.851235 21764 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:29:16.853914 21832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:29:16.855852 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 21764
I20260620 04:29:16.855996 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781929496330398-392-0/minicluster-data/master-0/wal/instance
I20260620 04:29:16.856312 392 external_mini_cluster.cc:1428] Setting key a733a2205fc333f8f414689c519ea9fd
I20260620 04:29:16.859776 21832 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7405de2f7254845a879225225a4e743: Bootstrap starting.
I20260620 04:29:16.861930 21832 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d7405de2f7254845a879225225a4e743: Neither blocks nor log segments found. Creating new log.
I20260620 04:29:16.862620 21832 log.cc:824] T 00000000000000000000000000000000 P d7405de2f7254845a879225225a4e743: Log is configured to *not* fsync() on all Append() calls
I20260620 04:29:16.864861 21832 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7405de2f7254845a879225225a4e743: No bootstrap required, opened a new log
I20260620 04:29:16.865284 21835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:16.857693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.32:59537 (local address 127.0.0.1:42181)
0620 04:29:16.858101 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:16.858118 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:16.858153 (+ 35us) server_negotiation.cc:407] Connection header received
0620 04:29:16.858900 (+ 747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:16.858917 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:16.859296 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:16.859594 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:29:16.860029 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:16.861001 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:16.861853 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:16.862360 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:16.862501 (+ 141us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:16.863371 (+ 870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:29:16.863378 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:29:16.863394 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:29:16.863857 (+ 463us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:29:16.863922 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:29:16.863951 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:29:16.864108 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 04:29:16.864261 (+ 153us) server_negotiation.cc:299] Negotiation successful
0620 04:29:16.864386 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":117,"threads_started":1}
I20260620 04:29:16.868186 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 04:29:16.868887 21832 raft_consensus.cc:348] T 00000000000000000000000000000000 P d7405de2f7254845a879225225a4e743 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7405de2f7254845a879225225a4e743" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42181 } }
I20260620 04:29:16.869158 21832 raft_consensus.cc:374] T 00000000000000000000000000000000 P d7405de2f7254845a879225225a4e743 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:29:16.869217 21832 raft_consensus.cc:729] T 00000000000000000000000000000000 P d7405de2f7254845a879225225a4e743 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7405de2f7254845a879225225a4e743, State: Initialized, Role: FOLLOWER
I20260620 04:29:16.869693 21832 consensus_queue.cc:260] T 00000000000000000000000000000000 P d7405de2f7254845a879225225a4e743 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7405de2f7254845a879225225a4e743" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42181 } }
I20260620 04:29:16.869849 21832 raft_consensus.cc:388] T 00000000000000000000000000000000 P d7405de2f7254845a879225225a4e743 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:29:16.869915 21832 raft_consensus.cc:482] T 00000000000000000000000000000000 P d7405de2f7254845a879225225a4e743 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:29:16.870011 21832 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d7405de2f7254845a879225225a4e743 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:29:16.871096 21832 raft_consensus.cc:504] T 00000000000000000000000000000000 P d7405de2f7254845a879225225a4e743 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7405de2f7254845a879225225a4e743" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42181 } }
I20260620 04:29:16.871414 21832 leader_election.cc:302] T 00000000000000000000000000000000 P d7405de2f7254845a879225225a4e743 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d7405de2f7254845a879225225a4e743; no voters:
I20260620 04:29:16.871701 21832 leader_election.cc:288] T 00000000000000000000000000000000 P d7405de2f7254845a879225225a4e743 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:29:16.871783 21842 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d7405de2f7254845a879225225a4e743 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:29:16.872000 21842 raft_consensus.cc:686] T 00000000000000000000000000000000 P d7405de2f7254845a879225225a4e743 [term 1 LEADER]: Becoming Leader. State: Replica: d7405de2f7254845a879225225a4e743, State: Running, Role: LEADER
I20260620 04:29:16.872344 21842 consensus_queue.cc:237] T 00000000000000000000000000000000 P d7405de2f7254845a879225225a4e743 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7405de2f7254845a879225225a4e743" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42181 } }
I20260620 04:29:16.873030 21832 sys_catalog.cc:564] T 00000000000000000000000000000000 P d7405de2f7254845a879225225a4e743 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:29:16.873540 21844 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7405de2f7254845a879225225a4e743 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d7405de2f7254845a879225225a4e743. Latest consensus state: current_term: 1 leader_uuid: "d7405de2f7254845a879225225a4e743" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7405de2f7254845a879225225a4e743" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42181 } } }
I20260620 04:29:16.873710 21844 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7405de2f7254845a879225225a4e743 [sys.catalog]: This master's current role is: LEADER
I20260620 04:29:16.873878 21843 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7405de2f7254845a879225225a4e743 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d7405de2f7254845a879225225a4e743" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7405de2f7254845a879225225a4e743" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42181 } } }
I20260620 04:29:16.873973 21843 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7405de2f7254845a879225225a4e743 [sys.catalog]: This master's current role is: LEADER
I20260620 04:29:16.875058 21835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:16.870254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.32:33845 (local address 127.0.0.1:42181)
0620 04:29:16.870369 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:16.870373 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:16.870548 (+ 175us) server_negotiation.cc:407] Connection header received
0620 04:29:16.870666 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:16.870668 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:16.870721 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:16.870780 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:29:16.871043 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:16.871597 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:16.872636 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:16.872967 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:16.873065 (+ 98us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:16.874365 (+ 1300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:29:16.874368 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:29:16.874371 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:29:16.874384 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:29:16.874410 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:29:16.874422 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:29:16.874502 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 04:29:16.874890 (+ 388us) server_negotiation.cc:299] Negotiation successful
0620 04:29:16.874926 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20260620 04:29:16.877790 21858 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d7405de2f7254845a879225225a4e743: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 04:29:16.877851 21858 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 04:29:16.877924 21846 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:29:16.878558 21846 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:29:16.883569 21846 catalog_manager.cc:1347] Generated new cluster ID: 609070aa7a304258aa0cf9864090ca18
I20260620 04:29:16.883625 21846 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:29:16.905625 21846 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:29:16.906921 21846 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:29:16.912853 21846 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d7405de2f7254845a879225225a4e743: Generated new TSK 0
I20260620 04:29:16.913535 21846 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 04:29:16.921726 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 21764
W20260620 04:29:16.926307 21749 connection.cc:535] client connection to 127.0.0.1:42181 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42181): Connection reset by peer (error 104)
2026-06-20T04:29:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1305 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781929496330398-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:29:16 dist-test-slave-nk6d krb5kdc[21863](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:29:16 dist-test-slave-nk6d krb5kdc[21863](info): set up 2 sockets
Jun 20 04:29:16 dist-test-slave-nk6d krb5kdc[21863](info): commencing operation
krb5kdc: starting...
W20260620 04:29:17.871933 392 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781929496330398-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781929496330398-392-0/krb5kdc/test-user.keytab.
Jun 20 04:29:17 dist-test-slave-nk6d krb5kdc[21863](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929757, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:29:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:29:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781929496330398-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781929496330398-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 04:29:17.985179 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781929496330398-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781929496330398-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781929496330398-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32981
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781929496330398-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32981
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:29:18.091053 21876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:29:18.091301 21876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:29:18.091341 21876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:29:18.095269 21876 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:29:18.095325 21876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:29:18.095345 21876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:29:18.095359 21876 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:29:18.095373 21876 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:29:18.100529 21876 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781929496330398-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781929496330398-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32981
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781929496330398-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32981
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21876
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781929496330398-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:29:18.101524 21876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:29:18.102491 21876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:29:18.110587 21882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:29:18.110594 21881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:29:18.110618 21876 server_base.cc:1034] running on GCE node
W20260620 04:29:18.110666 21884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:29:18.111364 21876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:29:18.112476 21876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:29:18.113649 21876 hybrid_clock.cc:648] HybridClock initialized: now 1781929758113628 us; error 34 us; skew 500 ppm
Jun 20 04:29:18 dist-test-slave-nk6d krb5kdc[21863](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929758, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:29:18.121479 21876 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 04:29:18.122366 21876 webserver.cc:460] Webserver started at http://127.0.0.1:34007/ using document root <none> and password file <none>
I20260620 04:29:18.122902 21876 fs_manager.cc:359] Metadata directory not provided
I20260620 04:29:18.122967 21876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:29:18.123158 21876 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:29:18.125175 21876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781929496330398-392-0/minicluster-data/master-0/data/instance:
uuid: "1ce47338981142fea2ca4352768c152e"
format_stamp: "Formatted at 2026-06-20 04:29:18 on dist-test-slave-nk6d"
server_key: "4488b1d670ba1a3740f0e3be77cab69a"
server_key_iv: "54fd6439004f26db61071fe084bbfbc9"
server_key_version: "encryptionkey@0"
I20260620 04:29:18.125689 21876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781929496330398-392-0/minicluster-data/master-0/wal/instance:
uuid: "1ce47338981142fea2ca4352768c152e"
format_stamp: "Formatted at 2026-06-20 04:29:18 on dist-test-slave-nk6d"
server_key: "4488b1d670ba1a3740f0e3be77cab69a"
server_key_iv: "54fd6439004f26db61071fe084bbfbc9"
server_key_version: "encryptionkey@0"
I20260620 04:29:18.129256 21876 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 04:29:18.131644 21891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:29:18.132949 21876 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:29:18.133075 21876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781929496330398-392-0/minicluster-data/master-0/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781929496330398-392-0/minicluster-data/master-0/wal
uuid: "1ce47338981142fea2ca4352768c152e"
format_stamp: "Formatted at 2026-06-20 04:29:18 on dist-test-slave-nk6d"
server_key: "4488b1d670ba1a3740f0e3be77cab69a"
server_key_iv: "54fd6439004f26db61071fe084bbfbc9"
server_key_version: "encryptionkey@0"
I20260620 04:29:18.133179 21876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781929496330398-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781929496330398-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781929496330398-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:29:18.137564 21876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:29:18.140662 21876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:29:18.140830 21876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:29:18.147456 21876 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32981
I20260620 04:29:18.147472 21943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32981 every 8 connection(s)
I20260620 04:29:18.148432 21876 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:29:18.150767 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 21876
I20260620 04:29:18.150892 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781929496330398-392-0/minicluster-data/master-0/wal/instance
I20260620 04:29:18.151185 392 external_mini_cluster.cc:1428] Setting key 6ea29bfc5a90301d6adac9945de09cb0
I20260620 04:29:18.151247 21944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:29:18.156514 21947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:18.152509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.32:43787 (local address 127.0.0.1:32981)
0620 04:29:18.152894 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:18.152909 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:18.152945 (+ 36us) server_negotiation.cc:407] Connection header received
0620 04:29:18.153840 (+ 895us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:18.153862 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:18.154141 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:18.154431 (+ 290us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:29:18.155247 (+ 816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:29:18.155251 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:29:18.155262 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:29:18.155273 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:29:18.155324 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:29:18.155342 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:29:18.155449 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 04:29:18.155602 (+ 153us) server_negotiation.cc:299] Negotiation successful
0620 04:29:18.155769 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":120,"threads_started":1}
I20260620 04:29:18.157728 21944 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ce47338981142fea2ca4352768c152e: Bootstrap starting.
I20260620 04:29:18.159411 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 04:29:18.160295 21944 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ce47338981142fea2ca4352768c152e: Neither blocks nor log segments found. Creating new log.
I20260620 04:29:18.161197 21944 log.cc:824] T 00000000000000000000000000000000 P 1ce47338981142fea2ca4352768c152e: Log is configured to *not* fsync() on all Append() calls
I20260620 04:29:18.163141 21947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:18.161635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.32:57075 (local address 127.0.0.1:32981)
0620 04:29:18.161735 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:18.161739 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:18.161934 (+ 195us) server_negotiation.cc:407] Connection header received
0620 04:29:18.162044 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:18.162047 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:18.162079 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:18.162140 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:29:18.162793 (+ 653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:29:18.162797 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:29:18.162799 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:29:18.162811 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:29:18.162836 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:29:18.162846 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:29:18.162916 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 04:29:18.162989 (+ 73us) server_negotiation.cc:299] Negotiation successful
0620 04:29:18.163027 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 04:29:18.163272 21944 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ce47338981142fea2ca4352768c152e: No bootstrap required, opened a new log
I20260620 04:29:18.166940 21944 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ce47338981142fea2ca4352768c152e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ce47338981142fea2ca4352768c152e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32981 } }
I20260620 04:29:18.167176 21944 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1ce47338981142fea2ca4352768c152e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:29:18.167227 21944 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ce47338981142fea2ca4352768c152e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ce47338981142fea2ca4352768c152e, State: Initialized, Role: FOLLOWER
I20260620 04:29:18.167644 21944 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ce47338981142fea2ca4352768c152e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ce47338981142fea2ca4352768c152e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32981 } }
I20260620 04:29:18.167771 21944 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1ce47338981142fea2ca4352768c152e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:29:18.167832 21944 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ce47338981142fea2ca4352768c152e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:29:18.167927 21944 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ce47338981142fea2ca4352768c152e [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:29:18.168941 21944 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ce47338981142fea2ca4352768c152e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ce47338981142fea2ca4352768c152e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32981 } }
I20260620 04:29:18.169258 21944 leader_election.cc:302] T 00000000000000000000000000000000 P 1ce47338981142fea2ca4352768c152e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ce47338981142fea2ca4352768c152e; no voters:
I20260620 04:29:18.169564 21944 leader_election.cc:288] T 00000000000000000000000000000000 P 1ce47338981142fea2ca4352768c152e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:29:18.169688 21954 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ce47338981142fea2ca4352768c152e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:29:18.169878 21954 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ce47338981142fea2ca4352768c152e [term 1 LEADER]: Becoming Leader. State: Replica: 1ce47338981142fea2ca4352768c152e, State: Running, Role: LEADER
I20260620 04:29:18.170189 21954 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ce47338981142fea2ca4352768c152e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ce47338981142fea2ca4352768c152e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32981 } }
I20260620 04:29:18.170660 21944 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ce47338981142fea2ca4352768c152e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:29:18.171672 21955 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ce47338981142fea2ca4352768c152e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1ce47338981142fea2ca4352768c152e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ce47338981142fea2ca4352768c152e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32981 } } }
I20260620 04:29:18.171804 21955 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ce47338981142fea2ca4352768c152e [sys.catalog]: This master's current role is: LEADER
I20260620 04:29:18.171999 21956 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ce47338981142fea2ca4352768c152e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ce47338981142fea2ca4352768c152e. Latest consensus state: current_term: 1 leader_uuid: "1ce47338981142fea2ca4352768c152e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ce47338981142fea2ca4352768c152e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32981 } } }
I20260620 04:29:18.172139 21956 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ce47338981142fea2ca4352768c152e [sys.catalog]: This master's current role is: LEADER
I20260620 04:29:18.172235 21963 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:29:18.175199 21963 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:29:18.180567 21963 catalog_manager.cc:1347] Generated new cluster ID: 1538916a5935493b999c75d937e6bd8c
I20260620 04:29:18.180634 21963 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:29:18.196265 21963 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:29:18.196599 21963 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:29:18.208000 21963 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1ce47338981142fea2ca4352768c152e: Generated new TSK 0
I20260620 04:29:18.208829 21963 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 04:29:18.213775 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 21876
2026-06-20T04:29:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1292 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781929496330398-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:29:18 dist-test-slave-nk6d krb5kdc[21975](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:29:18 dist-test-slave-nk6d krb5kdc[21975](info): set up 2 sockets
Jun 20 04:29:18 dist-test-slave-nk6d krb5kdc[21975](info): commencing operation
krb5kdc: starting...
W20260620 04:29:19.163877 392 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781929496330398-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781929496330398-392-0/krb5kdc/test-user.keytab.
Jun 20 04:29:19 dist-test-slave-nk6d krb5kdc[21975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:29:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:29:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 04:29:19.272491 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781929496330398-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781929496330398-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781929496330398-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32933
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:32933
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:29:19.377501 21988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:29:19.377781 21988 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:29:19.377821 21988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:29:19.381664 21988 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:29:19.381708 21988 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:29:19.381726 21988 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:29:19.381740 21988 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:29:19.381753 21988 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:29:19.386704 21988 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781929496330398-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781929496330398-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:32933
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32933
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21988
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781929496330398-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:29:19.387602 21988 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:29:19.388526 21988 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:29:19.396482 21994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:29:19.396471 21993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:29:19.396672 21988 server_base.cc:1034] running on GCE node
W20260620 04:29:19.396780 21996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:29:19.397292 21988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:29:19.398312 21988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:29:19.399454 21988 hybrid_clock.cc:648] HybridClock initialized: now 1781929759399432 us; error 39 us; skew 500 ppm
Jun 20 04:29:19 dist-test-slave-nk6d krb5kdc[21975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929759, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:29:19.402031 21988 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 04:29:19.402851 21988 webserver.cc:460] Webserver started at http://127.0.98.62:35765/ using document root <none> and password file <none>
I20260620 04:29:19.403350 21988 fs_manager.cc:359] Metadata directory not provided
I20260620 04:29:19.403411 21988 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:29:19.403622 21988 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:29:19.405571 21988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781929496330398-392-0/minicluster-data/master-0/data/instance:
uuid: "8998736b42d64cf88b922b76e9ca3c5f"
format_stamp: "Formatted at 2026-06-20 04:29:19 on dist-test-slave-nk6d"
server_key: "fbd2548d4aba8818014fddf40cb9e83e"
server_key_iv: "2cd8360039e90436ddfa5669375dccdb"
server_key_version: "encryptionkey@0"
I20260620 04:29:19.406075 21988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781929496330398-392-0/minicluster-data/master-0/wal/instance:
uuid: "8998736b42d64cf88b922b76e9ca3c5f"
format_stamp: "Formatted at 2026-06-20 04:29:19 on dist-test-slave-nk6d"
server_key: "fbd2548d4aba8818014fddf40cb9e83e"
server_key_iv: "2cd8360039e90436ddfa5669375dccdb"
server_key_version: "encryptionkey@0"
I20260620 04:29:19.409545 21988 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260620 04:29:19.411859 22003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:29:19.413043 21988 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260620 04:29:19.413149 21988 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781929496330398-392-0/minicluster-data/master-0/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781929496330398-392-0/minicluster-data/master-0/wal
uuid: "8998736b42d64cf88b922b76e9ca3c5f"
format_stamp: "Formatted at 2026-06-20 04:29:19 on dist-test-slave-nk6d"
server_key: "fbd2548d4aba8818014fddf40cb9e83e"
server_key_iv: "2cd8360039e90436ddfa5669375dccdb"
server_key_version: "encryptionkey@0"
I20260620 04:29:19.413236 21988 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781929496330398-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781929496330398-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781929496330398-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:29:19.436766 21988 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:29:19.439775 21988 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:29:19.439949 21988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:29:19.446605 21988 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:32933
I20260620 04:29:19.446628 22055 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:32933 every 8 connection(s)
I20260620 04:29:19.447515 21988 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:29:19.448027 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 21988
I20260620 04:29:19.448158 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781929496330398-392-0/minicluster-data/master-0/wal/instance
I20260620 04:29:19.448406 392 external_mini_cluster.cc:1428] Setting key d1f87ea76090a2322b65f7de2693c214
I20260620 04:29:19.450491 22056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:29:19.455646 22056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8998736b42d64cf88b922b76e9ca3c5f: Bootstrap starting.
Jun 20 04:29:19 dist-test-slave-nk6d krb5kdc[21975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:29:19.457842 22056 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8998736b42d64cf88b922b76e9ca3c5f: Neither blocks nor log segments found. Creating new log.
I20260620 04:29:19.458693 22056 log.cc:824] T 00000000000000000000000000000000 P 8998736b42d64cf88b922b76e9ca3c5f: Log is configured to *not* fsync() on all Append() calls
I20260620 04:29:19.460867 22056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8998736b42d64cf88b922b76e9ca3c5f: No bootstrap required, opened a new log
I20260620 04:29:19.461906 22059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:19.449716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56903 (local address 127.0.98.62:32933)
0620 04:29:19.450120 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:19.450134 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:19.450168 (+ 34us) server_negotiation.cc:407] Connection header received
0620 04:29:19.450819 (+ 651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:19.450837 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:19.451169 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:19.451467 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:29:19.452266 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:19.453244 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:19.454043 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:19.454491 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:19.454637 (+ 146us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:19.457117 (+ 2480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:29:19.457122 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:29:19.457134 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:29:19.457160 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:29:19.458892 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:29:19.459406 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:29:19.459412 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:29:19.459418 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:29:19.459550 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:29:19.459832 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:29:19.459835 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:29:19.459837 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:29:19.460336 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:29:19.460527 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0620 04:29:19.460795 (+ 268us) server_negotiation.cc:299] Negotiation successful
0620 04:29:19.461010 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":134,"threads_started":1}
I20260620 04:29:19.463912 22056 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8998736b42d64cf88b922b76e9ca3c5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8998736b42d64cf88b922b76e9ca3c5f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32933 } }
I20260620 04:29:19.464196 22056 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8998736b42d64cf88b922b76e9ca3c5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:29:19.464258 22056 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8998736b42d64cf88b922b76e9ca3c5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8998736b42d64cf88b922b76e9ca3c5f, State: Initialized, Role: FOLLOWER
I20260620 04:29:19.464676 22056 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8998736b42d64cf88b922b76e9ca3c5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8998736b42d64cf88b922b76e9ca3c5f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32933 } }
I20260620 04:29:19.464816 22056 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8998736b42d64cf88b922b76e9ca3c5f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:29:19.464884 22056 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8998736b42d64cf88b922b76e9ca3c5f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:29:19.464985 22056 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8998736b42d64cf88b922b76e9ca3c5f [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:29:19.465003 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 04:29:19.465866 22056 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8998736b42d64cf88b922b76e9ca3c5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8998736b42d64cf88b922b76e9ca3c5f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32933 } }
I20260620 04:29:19.466188 22056 leader_election.cc:302] T 00000000000000000000000000000000 P 8998736b42d64cf88b922b76e9ca3c5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8998736b42d64cf88b922b76e9ca3c5f; no voters:
I20260620 04:29:19.466463 22056 leader_election.cc:288] T 00000000000000000000000000000000 P 8998736b42d64cf88b922b76e9ca3c5f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:29:19.466539 22065 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8998736b42d64cf88b922b76e9ca3c5f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:29:19.466759 22065 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8998736b42d64cf88b922b76e9ca3c5f [term 1 LEADER]: Becoming Leader. State: Replica: 8998736b42d64cf88b922b76e9ca3c5f, State: Running, Role: LEADER
I20260620 04:29:19.467185 22065 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8998736b42d64cf88b922b76e9ca3c5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8998736b42d64cf88b922b76e9ca3c5f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32933 } }
I20260620 04:29:19.467710 22056 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8998736b42d64cf88b922b76e9ca3c5f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:29:19.468340 22065 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8998736b42d64cf88b922b76e9ca3c5f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8998736b42d64cf88b922b76e9ca3c5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8998736b42d64cf88b922b76e9ca3c5f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32933 } } }
I20260620 04:29:19.468514 22065 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8998736b42d64cf88b922b76e9ca3c5f [sys.catalog]: This master's current role is: LEADER
I20260620 04:29:19.468742 22065 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8998736b42d64cf88b922b76e9ca3c5f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8998736b42d64cf88b922b76e9ca3c5f. Latest consensus state: current_term: 1 leader_uuid: "8998736b42d64cf88b922b76e9ca3c5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8998736b42d64cf88b922b76e9ca3c5f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32933 } } }
I20260620 04:29:19.468825 22065 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8998736b42d64cf88b922b76e9ca3c5f [sys.catalog]: This master's current role is: LEADER
I20260620 04:29:19.469055 22075 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:29:19.472393 22075 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:29:19.477741 22059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:19.466785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59205 (local address 127.0.98.62:32933)
0620 04:29:19.466887 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:19.466891 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:19.469909 (+ 3018us) server_negotiation.cc:407] Connection header received
0620 04:29:19.470037 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:19.470040 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:19.470097 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:19.470173 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:29:19.470890 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:19.471621 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:19.472787 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:19.473120 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:19.473225 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:19.474728 (+ 1503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:29:19.474731 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:29:19.474733 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:29:19.474752 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:29:19.476008 (+ 1256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:29:19.476453 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:29:19.476457 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:29:19.476459 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:29:19.476545 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:29:19.476876 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:29:19.476879 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:29:19.476881 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:29:19.477092 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:29:19.477235 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 04:29:19.477498 (+ 263us) server_negotiation.cc:299] Negotiation successful
0620 04:29:19.477598 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 04:29:19.478207 22075 catalog_manager.cc:1347] Generated new cluster ID: dd3b52cd4a2e4dada4437a57f493eac2
I20260620 04:29:19.478258 22075 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:29:19.487383 22075 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:29:19.488286 22075 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:29:19.493503 22075 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8998736b42d64cf88b922b76e9ca3c5f: Generated new TSK 0
I20260620 04:29:19.494220 22075 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 04:29:19.505177 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 21988
W20260620 04:29:19.509598 21973 connection.cc:535] client connection to 127.0.98.62:32933 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:32933): Connection reset by peer (error 104)
2026-06-20T04:29:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1291 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781929496330398-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:29:19 dist-test-slave-nk6d krb5kdc[22087](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:29:19 dist-test-slave-nk6d krb5kdc[22087](info): set up 2 sockets
Jun 20 04:29:19 dist-test-slave-nk6d krb5kdc[22087](info): commencing operation
krb5kdc: starting...
W20260620 04:29:20.451794 392 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781929496330398-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781929496330398-392-0/krb5kdc/test-user.keytab.
Jun 20 04:29:20 dist-test-slave-nk6d krb5kdc[22087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929760, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:29:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:29:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 04:29:20.560859 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781929496330398-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781929496330398-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781929496330398-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44107
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44107
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:29:20.668828 22100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:29:20.669098 22100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:29:20.669144 22100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:29:20.673137 22100 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:29:20.673192 22100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:29:20.673210 22100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:29:20.673228 22100 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:29:20.673256 22100 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:29:20.678534 22100 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781929496330398-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781929496330398-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44107
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44107
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22100
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781929496330398-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:29:20.679534 22100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:29:20.680557 22100 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:29:20.688498 22105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:29:20.688480 22108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:29:20.688480 22106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:29:20.688570 22100 server_base.cc:1034] running on GCE node
I20260620 04:29:20.689250 22100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:29:20.690328 22100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:29:20.691478 22100 hybrid_clock.cc:648] HybridClock initialized: now 1781929760691468 us; error 27 us; skew 500 ppm
Jun 20 04:29:20 dist-test-slave-nk6d krb5kdc[22087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929760, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:29:20.694010 22100 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 04:29:20.694844 22100 webserver.cc:460] Webserver started at http://127.0.98.62:40621/ using document root <none> and password file <none>
I20260620 04:29:20.695365 22100 fs_manager.cc:359] Metadata directory not provided
I20260620 04:29:20.695420 22100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:29:20.695601 22100 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:29:20.697624 22100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781929496330398-392-0/minicluster-data/master-0/data/instance:
uuid: "49b7889cceb44a3e9d64b9fd31884d21"
format_stamp: "Formatted at 2026-06-20 04:29:20 on dist-test-slave-nk6d"
server_key: "85049ed1370b45334e5c2f8d74407ffd"
server_key_iv: "0357636a87560a04b4df43c521bfa5d9"
server_key_version: "encryptionkey@0"
I20260620 04:29:20.698150 22100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781929496330398-392-0/minicluster-data/master-0/wal/instance:
uuid: "49b7889cceb44a3e9d64b9fd31884d21"
format_stamp: "Formatted at 2026-06-20 04:29:20 on dist-test-slave-nk6d"
server_key: "85049ed1370b45334e5c2f8d74407ffd"
server_key_iv: "0357636a87560a04b4df43c521bfa5d9"
server_key_version: "encryptionkey@0"
I20260620 04:29:20.701757 22100 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 04:29:20.704044 22115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:29:20.705211 22100 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 04:29:20.705318 22100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781929496330398-392-0/minicluster-data/master-0/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781929496330398-392-0/minicluster-data/master-0/wal
uuid: "49b7889cceb44a3e9d64b9fd31884d21"
format_stamp: "Formatted at 2026-06-20 04:29:20 on dist-test-slave-nk6d"
server_key: "85049ed1370b45334e5c2f8d74407ffd"
server_key_iv: "0357636a87560a04b4df43c521bfa5d9"
server_key_version: "encryptionkey@0"
I20260620 04:29:20.705408 22100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781929496330398-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781929496330398-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781929496330398-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:29:20.727994 22100 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:29:20.730764 22100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:29:20.730929 22100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:29:20.737370 22100 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44107
I20260620 04:29:20.737399 22167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44107 every 8 connection(s)
I20260620 04:29:20.738251 22100 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:29:20.740913 22168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:29:20.746143 22168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49b7889cceb44a3e9d64b9fd31884d21: Bootstrap starting.
I20260620 04:29:20.746654 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 22100
I20260620 04:29:20.746766 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781929496330398-392-0/minicluster-data/master-0/wal/instance
I20260620 04:29:20.747006 392 external_mini_cluster.cc:1428] Setting key af2eb4fb1d216f19647605a75e6a55d7
I20260620 04:29:20.748873 22168 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 49b7889cceb44a3e9d64b9fd31884d21: Neither blocks nor log segments found. Creating new log.
I20260620 04:29:20.749733 22168 log.cc:824] T 00000000000000000000000000000000 P 49b7889cceb44a3e9d64b9fd31884d21: Log is configured to *not* fsync() on all Append() calls
I20260620 04:29:20.751821 22168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49b7889cceb44a3e9d64b9fd31884d21: No bootstrap required, opened a new log
I20260620 04:29:20.755937 22168 raft_consensus.cc:348] T 00000000000000000000000000000000 P 49b7889cceb44a3e9d64b9fd31884d21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49b7889cceb44a3e9d64b9fd31884d21" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44107 } }
I20260620 04:29:20.756235 22168 raft_consensus.cc:374] T 00000000000000000000000000000000 P 49b7889cceb44a3e9d64b9fd31884d21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:29:20.756289 22168 raft_consensus.cc:729] T 00000000000000000000000000000000 P 49b7889cceb44a3e9d64b9fd31884d21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49b7889cceb44a3e9d64b9fd31884d21, State: Initialized, Role: FOLLOWER
I20260620 04:29:20.756696 22171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:20.748390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43313 (local address 127.0.98.62:44107)
0620 04:29:20.748830 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:20.748849 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:20.748892 (+ 43us) server_negotiation.cc:407] Connection header received
0620 04:29:20.749743 (+ 851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:20.749768 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:20.750202 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:20.750516 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:29:20.750925 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:20.752314 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:20.753171 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:20.753860 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:20.754021 (+ 161us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:20.754724 (+ 703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:29:20.754731 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:29:20.754747 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:29:20.755251 (+ 504us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:29:20.755314 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:29:20.755342 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:29:20.755469 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 04:29:20.755602 (+ 133us) server_negotiation.cc:299] Negotiation successful
0620 04:29:20.755728 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":124,"threads_started":1}
I20260620 04:29:20.756752 22168 consensus_queue.cc:260] T 00000000000000000000000000000000 P 49b7889cceb44a3e9d64b9fd31884d21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49b7889cceb44a3e9d64b9fd31884d21" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44107 } }
I20260620 04:29:20.756925 22168 raft_consensus.cc:388] T 00000000000000000000000000000000 P 49b7889cceb44a3e9d64b9fd31884d21 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:29:20.756992 22168 raft_consensus.cc:482] T 00000000000000000000000000000000 P 49b7889cceb44a3e9d64b9fd31884d21 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:29:20.757087 22168 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 49b7889cceb44a3e9d64b9fd31884d21 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:29:20.757900 22168 raft_consensus.cc:504] T 00000000000000000000000000000000 P 49b7889cceb44a3e9d64b9fd31884d21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49b7889cceb44a3e9d64b9fd31884d21" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44107 } }
I20260620 04:29:20.758208 22168 leader_election.cc:302] T 00000000000000000000000000000000 P 49b7889cceb44a3e9d64b9fd31884d21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 49b7889cceb44a3e9d64b9fd31884d21; no voters:
I20260620 04:29:20.758527 22168 leader_election.cc:288] T 00000000000000000000000000000000 P 49b7889cceb44a3e9d64b9fd31884d21 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:29:20.758572 22173 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 49b7889cceb44a3e9d64b9fd31884d21 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:29:20.758747 22173 raft_consensus.cc:686] T 00000000000000000000000000000000 P 49b7889cceb44a3e9d64b9fd31884d21 [term 1 LEADER]: Becoming Leader. State: Replica: 49b7889cceb44a3e9d64b9fd31884d21, State: Running, Role: LEADER
I20260620 04:29:20.759070 22173 consensus_queue.cc:237] T 00000000000000000000000000000000 P 49b7889cceb44a3e9d64b9fd31884d21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49b7889cceb44a3e9d64b9fd31884d21" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44107 } }
I20260620 04:29:20.759446 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 04:29:20.759723 22168 sys_catalog.cc:564] T 00000000000000000000000000000000 P 49b7889cceb44a3e9d64b9fd31884d21 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:29:20.760788 22174 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49b7889cceb44a3e9d64b9fd31884d21 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "49b7889cceb44a3e9d64b9fd31884d21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49b7889cceb44a3e9d64b9fd31884d21" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44107 } } }
I20260620 04:29:20.760807 22175 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49b7889cceb44a3e9d64b9fd31884d21 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 49b7889cceb44a3e9d64b9fd31884d21. Latest consensus state: current_term: 1 leader_uuid: "49b7889cceb44a3e9d64b9fd31884d21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49b7889cceb44a3e9d64b9fd31884d21" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44107 } } }
I20260620 04:29:20.760926 22174 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49b7889cceb44a3e9d64b9fd31884d21 [sys.catalog]: This master's current role is: LEADER
I20260620 04:29:20.760926 22175 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49b7889cceb44a3e9d64b9fd31884d21 [sys.catalog]: This master's current role is: LEADER
I20260620 04:29:20.761339 22186 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:29:20.764580 22186 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:29:20.766466 22171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:20.761639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51413 (local address 127.0.98.62:44107)
0620 04:29:20.761738 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:20.761742 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:20.762182 (+ 440us) server_negotiation.cc:407] Connection header received
0620 04:29:20.762375 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:20.762379 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:20.762426 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:20.762505 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:29:20.762809 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:20.763543 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:20.764636 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:20.764960 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:20.765066 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:20.765948 (+ 882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:29:20.765953 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:29:20.765956 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:29:20.765968 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:29:20.765997 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:29:20.766011 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:29:20.766091 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 04:29:20.766310 (+ 219us) server_negotiation.cc:299] Negotiation successful
0620 04:29:20.766345 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 04:29:20.770303 22186 catalog_manager.cc:1347] Generated new cluster ID: f86c47eb4c9449ef9c6226a826de1706
I20260620 04:29:20.770358 22186 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:29:20.797987 22186 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:29:20.799301 22186 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:29:20.805017 22186 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 49b7889cceb44a3e9d64b9fd31884d21: Generated new TSK 0
I20260620 04:29:20.805811 22186 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 04:29:20.812923 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 22100
W20260620 04:29:20.817852 22085 connection.cc:535] client connection to 127.0.98.62:44107 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:44107): Connection reset by peer (error 104)
2026-06-20T04:29:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1308 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781929496330398-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:29:20 dist-test-slave-nk6d krb5kdc[22199](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:29:20 dist-test-slave-nk6d krb5kdc[22199](info): set up 2 sockets
Jun 20 04:29:20 dist-test-slave-nk6d krb5kdc[22199](info): commencing operation
krb5kdc: starting...
W20260620 04:29:21.763891 392 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781929496330398-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781929496330398-392-0/krb5kdc/test-user.keytab.
Jun 20 04:29:21 dist-test-slave-nk6d krb5kdc[22199](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929761, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:29:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:29:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 04:29:21.872604 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781929496330398-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781929496330398-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781929496330398-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37113
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37113
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:29:21.976907 22212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:29:21.977181 22212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:29:21.977240 22212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:29:21.981176 22212 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:29:21.981230 22212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:29:21.981257 22212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:29:21.981280 22212 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:29:21.981307 22212 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:29:21.986328 22212 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781929496330398-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781929496330398-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37113
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37113
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22212
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781929496330398-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:29:21.987303 22212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:29:21.988286 22212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:29:21.996031 22220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:29:21.996044 22218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:29:21.996079 22217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:29:21.996202 22212 server_base.cc:1034] running on GCE node
I20260620 04:29:21.996793 22212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:29:21.997867 22212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:29:21.999027 22212 hybrid_clock.cc:648] HybridClock initialized: now 1781929761999009 us; error 28 us; skew 500 ppm
Jun 20 04:29:21 dist-test-slave-nk6d krb5kdc[22199](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929761, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:29:22.001637 22212 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 04:29:22.002446 22212 webserver.cc:460] Webserver started at http://127.0.98.62:40703/ using document root <none> and password file <none>
I20260620 04:29:22.002946 22212 fs_manager.cc:359] Metadata directory not provided
I20260620 04:29:22.003010 22212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:29:22.003221 22212 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:29:22.005223 22212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781929496330398-392-0/minicluster-data/master-0/data/instance:
uuid: "5e685a5d9aaa422d9abd96450319cc6c"
format_stamp: "Formatted at 2026-06-20 04:29:21 on dist-test-slave-nk6d"
server_key: "8492669521d642614231d2559c19d170"
server_key_iv: "f3693bca62a29e643e17b22c69441489"
server_key_version: "encryptionkey@0"
I20260620 04:29:22.005745 22212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781929496330398-392-0/minicluster-data/master-0/wal/instance:
uuid: "5e685a5d9aaa422d9abd96450319cc6c"
format_stamp: "Formatted at 2026-06-20 04:29:21 on dist-test-slave-nk6d"
server_key: "8492669521d642614231d2559c19d170"
server_key_iv: "f3693bca62a29e643e17b22c69441489"
server_key_version: "encryptionkey@0"
I20260620 04:29:22.009363 22212 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 04:29:22.011781 22227 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:29:22.013127 22212 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:29:22.013273 22212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781929496330398-392-0/minicluster-data/master-0/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781929496330398-392-0/minicluster-data/master-0/wal
uuid: "5e685a5d9aaa422d9abd96450319cc6c"
format_stamp: "Formatted at 2026-06-20 04:29:21 on dist-test-slave-nk6d"
server_key: "8492669521d642614231d2559c19d170"
server_key_iv: "f3693bca62a29e643e17b22c69441489"
server_key_version: "encryptionkey@0"
I20260620 04:29:22.013381 22212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781929496330398-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781929496330398-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781929496330398-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:29:22.017877 22212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:29:22.020857 22212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:29:22.021023 22212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:29:22.027601 22212 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37113
I20260620 04:29:22.027642 22279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37113 every 8 connection(s)
I20260620 04:29:22.028563 22212 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:29:22.031293 22280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:29:22.036940 22280 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e685a5d9aaa422d9abd96450319cc6c: Bootstrap starting.
I20260620 04:29:22.037730 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 22212
I20260620 04:29:22.037851 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781929496330398-392-0/minicluster-data/master-0/wal/instance
I20260620 04:29:22.038100 392 external_mini_cluster.cc:1428] Setting key aeb84cbf0bfc684b681bf87fb633fb5a
I20260620 04:29:22.039661 22280 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e685a5d9aaa422d9abd96450319cc6c: Neither blocks nor log segments found. Creating new log.
I20260620 04:29:22.040547 22280 log.cc:824] T 00000000000000000000000000000000 P 5e685a5d9aaa422d9abd96450319cc6c: Log is configured to *not* fsync() on all Append() calls
I20260620 04:29:22.042686 22280 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e685a5d9aaa422d9abd96450319cc6c: No bootstrap required, opened a new log
I20260620 04:29:22.043262 22283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:22.039200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41467 (local address 127.0.98.62:37113)
0620 04:29:22.039673 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:22.039692 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:22.039735 (+ 43us) server_negotiation.cc:407] Connection header received
0620 04:29:22.040600 (+ 865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:22.040624 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:22.040993 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:22.041296 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:29:22.041916 (+ 620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:29:22.041922 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:29:22.041936 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:29:22.041946 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:29:22.042013 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:29:22.042038 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:29:22.042146 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 04:29:22.042209 (+ 63us) server_negotiation.cc:299] Negotiation successful
0620 04:29:22.042353 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":132,"threads_started":1}
I20260620 04:29:22.045917 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 04:29:22.046372 22280 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e685a5d9aaa422d9abd96450319cc6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e685a5d9aaa422d9abd96450319cc6c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37113 } }
I20260620 04:29:22.046602 22280 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5e685a5d9aaa422d9abd96450319cc6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:29:22.046658 22280 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e685a5d9aaa422d9abd96450319cc6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e685a5d9aaa422d9abd96450319cc6c, State: Initialized, Role: FOLLOWER
I20260620 04:29:22.047086 22280 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e685a5d9aaa422d9abd96450319cc6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e685a5d9aaa422d9abd96450319cc6c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37113 } }
I20260620 04:29:22.047241 22280 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5e685a5d9aaa422d9abd96450319cc6c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:29:22.047307 22280 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5e685a5d9aaa422d9abd96450319cc6c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:29:22.047405 22280 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e685a5d9aaa422d9abd96450319cc6c [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:29:22.048501 22280 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5e685a5d9aaa422d9abd96450319cc6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e685a5d9aaa422d9abd96450319cc6c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37113 } }
I20260620 04:29:22.048837 22280 leader_election.cc:302] T 00000000000000000000000000000000 P 5e685a5d9aaa422d9abd96450319cc6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5e685a5d9aaa422d9abd96450319cc6c; no voters:
I20260620 04:29:22.049125 22280 leader_election.cc:288] T 00000000000000000000000000000000 P 5e685a5d9aaa422d9abd96450319cc6c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:29:22.049234 22290 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5e685a5d9aaa422d9abd96450319cc6c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:29:22.049400 22290 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5e685a5d9aaa422d9abd96450319cc6c [term 1 LEADER]: Becoming Leader. State: Replica: 5e685a5d9aaa422d9abd96450319cc6c, State: Running, Role: LEADER
I20260620 04:29:22.049744 22283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:22.047902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49547 (local address 127.0.98.62:37113)
0620 04:29:22.048051 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:22.048056 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:22.048229 (+ 173us) server_negotiation.cc:407] Connection header received
0620 04:29:22.048357 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:22.048360 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:22.048393 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:22.048455 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:29:22.049194 (+ 739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:29:22.049197 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:29:22.049200 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:29:22.049212 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:29:22.049241 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:29:22.049253 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:29:22.049330 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 04:29:22.049587 (+ 257us) server_negotiation.cc:299] Negotiation successful
0620 04:29:22.049629 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 04:29:22.049832 22290 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e685a5d9aaa422d9abd96450319cc6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e685a5d9aaa422d9abd96450319cc6c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37113 } }
I20260620 04:29:22.050493 22280 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e685a5d9aaa422d9abd96450319cc6c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:29:22.051100 22290 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e685a5d9aaa422d9abd96450319cc6c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e685a5d9aaa422d9abd96450319cc6c. Latest consensus state: current_term: 1 leader_uuid: "5e685a5d9aaa422d9abd96450319cc6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e685a5d9aaa422d9abd96450319cc6c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37113 } } }
I20260620 04:29:22.051220 22290 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e685a5d9aaa422d9abd96450319cc6c [sys.catalog]: This master's current role is: LEADER
I20260620 04:29:22.051661 22291 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e685a5d9aaa422d9abd96450319cc6c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5e685a5d9aaa422d9abd96450319cc6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e685a5d9aaa422d9abd96450319cc6c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37113 } } }
I20260620 04:29:22.051760 22291 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e685a5d9aaa422d9abd96450319cc6c [sys.catalog]: This master's current role is: LEADER
W20260620 04:29:22.055310 22305 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5e685a5d9aaa422d9abd96450319cc6c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 04:29:22.055375 22305 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 04:29:22.055441 22297 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:29:22.056190 22297 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:29:22.061462 22297 catalog_manager.cc:1347] Generated new cluster ID: 5c3e791096cb429abad3d0b611a94793
I20260620 04:29:22.061539 22297 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:29:22.081525 22297 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:29:22.081838 22297 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:29:22.089169 22297 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5e685a5d9aaa422d9abd96450319cc6c: Generated new TSK 0
I20260620 04:29:22.089876 22297 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 04:29:22.092590 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 22212
2026-06-20T04:29:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1279 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781929496330398-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:29:22 dist-test-slave-nk6d krb5kdc[22311](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:29:22 dist-test-slave-nk6d krb5kdc[22311](info): set up 2 sockets
Jun 20 04:29:22 dist-test-slave-nk6d krb5kdc[22311](info): commencing operation
krb5kdc: starting...
W20260620 04:29:23.040038 392 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781929496330398-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781929496330398-392-0/krb5kdc/test-user.keytab.
Jun 20 04:29:23 dist-test-slave-nk6d krb5kdc[22311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929763, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:29:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:29:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 04:29:23.150676 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781929496330398-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781929496330398-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781929496330398-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41875
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41875
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:29:23.258416 22324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:29:23.258671 22324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:29:23.258730 22324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:29:23.262737 22324 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:29:23.262812 22324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:29:23.262840 22324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:29:23.262862 22324 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:29:23.262892 22324 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:29:23.268127 22324 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781929496330398-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781929496330398-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41875
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41875
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22324
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781929496330398-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:29:23.269127 22324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:29:23.270118 22324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:29:23.279007 22330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:29:23.279024 22329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:29:23.279008 22332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:29:23.279439 22324 server_base.cc:1034] running on GCE node
I20260620 04:29:23.280077 22324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:29:23.281414 22324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:29:23.282583 22324 hybrid_clock.cc:648] HybridClock initialized: now 1781929763282547 us; error 51 us; skew 500 ppm
Jun 20 04:29:23 dist-test-slave-nk6d krb5kdc[22311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929763, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:29:23.285717 22324 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 04:29:23.286722 22324 webserver.cc:460] Webserver started at http://127.0.98.62:44095/ using document root <none> and password file <none>
I20260620 04:29:23.287364 22324 fs_manager.cc:359] Metadata directory not provided
I20260620 04:29:23.287436 22324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:29:23.287636 22324 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:29:23.289921 22324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781929496330398-392-0/minicluster-data/master-0/data/instance:
uuid: "20839686c63e4cf28e46a3b46e733654"
format_stamp: "Formatted at 2026-06-20 04:29:23 on dist-test-slave-nk6d"
server_key: "4d80b9352317d31ca26b488cd4a112e9"
server_key_iv: "d4d28c755ce406880341826f278924b8"
server_key_version: "encryptionkey@0"
I20260620 04:29:23.290586 22324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781929496330398-392-0/minicluster-data/master-0/wal/instance:
uuid: "20839686c63e4cf28e46a3b46e733654"
format_stamp: "Formatted at 2026-06-20 04:29:23 on dist-test-slave-nk6d"
server_key: "4d80b9352317d31ca26b488cd4a112e9"
server_key_iv: "d4d28c755ce406880341826f278924b8"
server_key_version: "encryptionkey@0"
I20260620 04:29:23.294987 22324 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260620 04:29:23.298059 22339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:29:23.299466 22324 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 04:29:23.299569 22324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781929496330398-392-0/minicluster-data/master-0/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781929496330398-392-0/minicluster-data/master-0/wal
uuid: "20839686c63e4cf28e46a3b46e733654"
format_stamp: "Formatted at 2026-06-20 04:29:23 on dist-test-slave-nk6d"
server_key: "4d80b9352317d31ca26b488cd4a112e9"
server_key_iv: "d4d28c755ce406880341826f278924b8"
server_key_version: "encryptionkey@0"
I20260620 04:29:23.299677 22324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781929496330398-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781929496330398-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781929496330398-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:29:23.325115 22324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:29:23.328238 22324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:29:23.328418 22324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:29:23.334818 22324 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41875
I20260620 04:29:23.334846 22391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41875 every 8 connection(s)
I20260620 04:29:23.335722 22324 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:29:23.336421 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 22324
I20260620 04:29:23.336563 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781929496330398-392-0/minicluster-data/master-0/wal/instance
I20260620 04:29:23.336848 392 external_mini_cluster.cc:1428] Setting key 67aa931f093df936884162a6fe8b38c3
I20260620 04:29:23.338949 22392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:29:23.345738 22392 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20839686c63e4cf28e46a3b46e733654: Bootstrap starting.
Jun 20 04:29:23 dist-test-slave-nk6d krb5kdc[22311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929763, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:29:23.348412 22392 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 20839686c63e4cf28e46a3b46e733654: Neither blocks nor log segments found. Creating new log.
I20260620 04:29:23.349318 22392 log.cc:824] T 00000000000000000000000000000000 P 20839686c63e4cf28e46a3b46e733654: Log is configured to *not* fsync() on all Append() calls
I20260620 04:29:23.351608 22392 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20839686c63e4cf28e46a3b46e733654: No bootstrap required, opened a new log
I20260620 04:29:23.351936 22395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:23.338228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39261 (local address 127.0.98.62:41875)
0620 04:29:23.338655 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:23.338670 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:23.338704 (+ 34us) server_negotiation.cc:407] Connection header received
0620 04:29:23.339385 (+ 681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:23.339402 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:23.339733 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:23.340077 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:29:23.341085 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:23.342092 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:23.343255 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:23.343717 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:23.343824 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:23.346692 (+ 2868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:29:23.346699 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:29:23.346717 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:29:23.346764 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:29:23.348888 (+ 2124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:29:23.349299 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:29:23.349306 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:29:23.349312 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:29:23.349455 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:29:23.349711 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:29:23.349715 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:29:23.349724 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:29:23.350211 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:29:23.350400 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0620 04:29:23.350651 (+ 251us) server_negotiation.cc:299] Negotiation successful
0620 04:29:23.350964 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":148,"threads_started":1}
I20260620 04:29:23.354947 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 04:29:23.356004 22392 raft_consensus.cc:348] T 00000000000000000000000000000000 P 20839686c63e4cf28e46a3b46e733654 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20839686c63e4cf28e46a3b46e733654" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41875 } }
I20260620 04:29:23.356262 22392 raft_consensus.cc:374] T 00000000000000000000000000000000 P 20839686c63e4cf28e46a3b46e733654 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:29:23.356330 22392 raft_consensus.cc:729] T 00000000000000000000000000000000 P 20839686c63e4cf28e46a3b46e733654 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20839686c63e4cf28e46a3b46e733654, State: Initialized, Role: FOLLOWER
I20260620 04:29:23.356763 22392 consensus_queue.cc:260] T 00000000000000000000000000000000 P 20839686c63e4cf28e46a3b46e733654 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20839686c63e4cf28e46a3b46e733654" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41875 } }
I20260620 04:29:23.356918 22392 raft_consensus.cc:388] T 00000000000000000000000000000000 P 20839686c63e4cf28e46a3b46e733654 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:29:23.356983 22392 raft_consensus.cc:482] T 00000000000000000000000000000000 P 20839686c63e4cf28e46a3b46e733654 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:29:23.357076 22392 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 20839686c63e4cf28e46a3b46e733654 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:29:23.358088 22392 raft_consensus.cc:504] T 00000000000000000000000000000000 P 20839686c63e4cf28e46a3b46e733654 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20839686c63e4cf28e46a3b46e733654" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41875 } }
I20260620 04:29:23.358456 22392 leader_election.cc:302] T 00000000000000000000000000000000 P 20839686c63e4cf28e46a3b46e733654 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 20839686c63e4cf28e46a3b46e733654; no voters:
I20260620 04:29:23.358806 22392 leader_election.cc:288] T 00000000000000000000000000000000 P 20839686c63e4cf28e46a3b46e733654 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:29:23.358907 22402 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 20839686c63e4cf28e46a3b46e733654 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:29:23.359114 22402 raft_consensus.cc:686] T 00000000000000000000000000000000 P 20839686c63e4cf28e46a3b46e733654 [term 1 LEADER]: Becoming Leader. State: Replica: 20839686c63e4cf28e46a3b46e733654, State: Running, Role: LEADER
I20260620 04:29:23.359421 22402 consensus_queue.cc:237] T 00000000000000000000000000000000 P 20839686c63e4cf28e46a3b46e733654 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20839686c63e4cf28e46a3b46e733654" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41875 } }
I20260620 04:29:23.360098 22392 sys_catalog.cc:564] T 00000000000000000000000000000000 P 20839686c63e4cf28e46a3b46e733654 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:29:23.360570 22402 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20839686c63e4cf28e46a3b46e733654 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 20839686c63e4cf28e46a3b46e733654. Latest consensus state: current_term: 1 leader_uuid: "20839686c63e4cf28e46a3b46e733654" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20839686c63e4cf28e46a3b46e733654" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41875 } } }
I20260620 04:29:23.360684 22402 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20839686c63e4cf28e46a3b46e733654 [sys.catalog]: This master's current role is: LEADER
I20260620 04:29:23.360956 22404 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20839686c63e4cf28e46a3b46e733654 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "20839686c63e4cf28e46a3b46e733654" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20839686c63e4cf28e46a3b46e733654" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41875 } } }
I20260620 04:29:23.361057 22404 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20839686c63e4cf28e46a3b46e733654 [sys.catalog]: This master's current role is: LEADER
W20260620 04:29:23.365000 22417 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 20839686c63e4cf28e46a3b46e733654: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 04:29:23.365083 22417 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 04:29:23.365162 22407 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:29:23.365357 22395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:23.356804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50333 (local address 127.0.98.62:41875)
0620 04:29:23.356925 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:23.356929 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:23.357148 (+ 219us) server_negotiation.cc:407] Connection header received
0620 04:29:23.357277 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:23.357280 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:23.357329 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:23.357411 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:29:23.357989 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:23.358723 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:23.359842 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:23.360229 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:23.360329 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:23.361887 (+ 1558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:29:23.361891 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:29:23.361894 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:29:23.361919 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:29:23.363698 (+ 1779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:29:23.364190 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:29:23.364194 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:29:23.364196 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:29:23.364320 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:29:23.364575 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:29:23.364578 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:29:23.364580 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:29:23.364753 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:29:23.364868 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 04:29:23.365089 (+ 221us) server_negotiation.cc:299] Negotiation successful
0620 04:29:23.365198 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 04:29:23.366180 22407 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:29:23.371451 22407 catalog_manager.cc:1347] Generated new cluster ID: 1d15a859ad3a411fb13938014a80e327
I20260620 04:29:23.371517 22407 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:29:23.380575 22407 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:29:23.381512 22407 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:29:23.386729 22407 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 20839686c63e4cf28e46a3b46e733654: Generated new TSK 0
I20260620 04:29:23.387351 22407 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 04:29:23.395409 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 22324
2026-06-20T04:29:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1303 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781929496330398-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:29:23 dist-test-slave-nk6d krb5kdc[22423](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:29:23 dist-test-slave-nk6d krb5kdc[22423](info): set up 2 sockets
Jun 20 04:29:23 dist-test-slave-nk6d krb5kdc[22423](info): commencing operation
krb5kdc: starting...
W20260620 04:29:24.335832 392 mini_kdc.cc:121] Time spent starting KDC: real 0.922s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781929496330398-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781929496330398-392-0/krb5kdc/test-user.keytab.
Jun 20 04:29:24 dist-test-slave-nk6d krb5kdc[22423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929764, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:29:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:29:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 04:29:24.444284 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781929496330398-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781929496330398-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781929496330398-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39143
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39143
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:29:24.562481 22436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:29:24.562721 22436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:29:24.562767 22436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:29:24.566751 22436 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:29:24.566808 22436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:29:24.566826 22436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:29:24.566841 22436 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:29:24.566856 22436 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:29:24.572335 22436 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781929496330398-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781929496330398-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39143
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39143
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22436
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781929496330398-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:29:24.573395 22436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:29:24.574405 22436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:29:24.582652 22444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:29:24.582684 22441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:29:24.582672 22442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:29:24.582738 22436 server_base.cc:1034] running on GCE node
I20260620 04:29:24.583433 22436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:29:24.584527 22436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:29:24.585717 22436 hybrid_clock.cc:648] HybridClock initialized: now 1781929764585691 us; error 38 us; skew 500 ppm
Jun 20 04:29:24 dist-test-slave-nk6d krb5kdc[22423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929764, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:29:24.588407 22436 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 04:29:24.589239 22436 webserver.cc:460] Webserver started at http://127.0.98.62:41289/ using document root <none> and password file <none>
I20260620 04:29:24.589831 22436 fs_manager.cc:359] Metadata directory not provided
I20260620 04:29:24.589903 22436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:29:24.590093 22436 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:29:24.592209 22436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781929496330398-392-0/minicluster-data/master-0/data/instance:
uuid: "f4c28652469049ca8d9d3a70daf9ed47"
format_stamp: "Formatted at 2026-06-20 04:29:24 on dist-test-slave-nk6d"
server_key: "d8e49f109dbf363ed69b2533440545ae"
server_key_iv: "daf35f55a8c6dae0f8aa2bf58bb2139e"
server_key_version: "encryptionkey@0"
I20260620 04:29:24.592923 22436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781929496330398-392-0/minicluster-data/master-0/wal/instance:
uuid: "f4c28652469049ca8d9d3a70daf9ed47"
format_stamp: "Formatted at 2026-06-20 04:29:24 on dist-test-slave-nk6d"
server_key: "d8e49f109dbf363ed69b2533440545ae"
server_key_iv: "daf35f55a8c6dae0f8aa2bf58bb2139e"
server_key_version: "encryptionkey@0"
I20260620 04:29:24.596506 22436 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:29:24.598827 22451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:29:24.600144 22436 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:29:24.600251 22436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781929496330398-392-0/minicluster-data/master-0/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781929496330398-392-0/minicluster-data/master-0/wal
uuid: "f4c28652469049ca8d9d3a70daf9ed47"
format_stamp: "Formatted at 2026-06-20 04:29:24 on dist-test-slave-nk6d"
server_key: "d8e49f109dbf363ed69b2533440545ae"
server_key_iv: "daf35f55a8c6dae0f8aa2bf58bb2139e"
server_key_version: "encryptionkey@0"
I20260620 04:29:24.600342 22436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781929496330398-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781929496330398-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781929496330398-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:29:24.655438 22436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:29:24.658298 22436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:29:24.658469 22436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:29:24.665088 22436 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39143
I20260620 04:29:24.665103 22503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39143 every 8 connection(s)
I20260620 04:29:24.666031 22436 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:29:24.668715 22504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:29:24.670141 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 22436
I20260620 04:29:24.670265 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781929496330398-392-0/minicluster-data/master-0/wal/instance
I20260620 04:29:24.670511 392 external_mini_cluster.cc:1428] Setting key f2ceb53ab7951c14fcb10f196e2f6f84
I20260620 04:29:24.674408 22504 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4c28652469049ca8d9d3a70daf9ed47: Bootstrap starting.
I20260620 04:29:24.676605 22504 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f4c28652469049ca8d9d3a70daf9ed47: Neither blocks nor log segments found. Creating new log.
I20260620 04:29:24.677282 22504 log.cc:824] T 00000000000000000000000000000000 P f4c28652469049ca8d9d3a70daf9ed47: Log is configured to *not* fsync() on all Append() calls
I20260620 04:29:24.678864 22507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:24.671818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40197 (local address 127.0.98.62:39143)
0620 04:29:24.672245 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:24.672259 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:24.672295 (+ 36us) server_negotiation.cc:407] Connection header received
0620 04:29:24.672931 (+ 636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:24.672948 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:24.673288 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:24.673605 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:29:24.673992 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:24.675038 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:24.675904 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:24.676401 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:24.676543 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:24.677290 (+ 747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:29:24.677295 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:29:24.677313 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:29:24.677707 (+ 394us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:29:24.677759 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:29:24.677782 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:29:24.677873 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 04:29:24.677994 (+ 121us) server_negotiation.cc:299] Negotiation successful
0620 04:29:24.678186 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":152,"threads_started":1}
I20260620 04:29:24.679019 22504 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4c28652469049ca8d9d3a70daf9ed47: No bootstrap required, opened a new log
I20260620 04:29:24.681483 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 04:29:24.682597 22504 raft_consensus.cc:348] T 00000000000000000000000000000000 P f4c28652469049ca8d9d3a70daf9ed47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4c28652469049ca8d9d3a70daf9ed47" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39143 } }
I20260620 04:29:24.682884 22504 raft_consensus.cc:374] T 00000000000000000000000000000000 P f4c28652469049ca8d9d3a70daf9ed47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:29:24.682942 22504 raft_consensus.cc:729] T 00000000000000000000000000000000 P f4c28652469049ca8d9d3a70daf9ed47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4c28652469049ca8d9d3a70daf9ed47, State: Initialized, Role: FOLLOWER
I20260620 04:29:24.683385 22504 consensus_queue.cc:260] T 00000000000000000000000000000000 P f4c28652469049ca8d9d3a70daf9ed47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4c28652469049ca8d9d3a70daf9ed47" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39143 } }
I20260620 04:29:24.683526 22504 raft_consensus.cc:388] T 00000000000000000000000000000000 P f4c28652469049ca8d9d3a70daf9ed47 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:29:24.683589 22504 raft_consensus.cc:482] T 00000000000000000000000000000000 P f4c28652469049ca8d9d3a70daf9ed47 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:29:24.683682 22504 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f4c28652469049ca8d9d3a70daf9ed47 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:29:24.684549 22504 raft_consensus.cc:504] T 00000000000000000000000000000000 P f4c28652469049ca8d9d3a70daf9ed47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4c28652469049ca8d9d3a70daf9ed47" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39143 } }
I20260620 04:29:24.684887 22504 leader_election.cc:302] T 00000000000000000000000000000000 P f4c28652469049ca8d9d3a70daf9ed47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f4c28652469049ca8d9d3a70daf9ed47; no voters:
I20260620 04:29:24.685228 22504 leader_election.cc:288] T 00000000000000000000000000000000 P f4c28652469049ca8d9d3a70daf9ed47 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:29:24.685323 22514 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f4c28652469049ca8d9d3a70daf9ed47 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:29:24.685535 22514 raft_consensus.cc:686] T 00000000000000000000000000000000 P f4c28652469049ca8d9d3a70daf9ed47 [term 1 LEADER]: Becoming Leader. State: Replica: f4c28652469049ca8d9d3a70daf9ed47, State: Running, Role: LEADER
I20260620 04:29:24.685837 22514 consensus_queue.cc:237] T 00000000000000000000000000000000 P f4c28652469049ca8d9d3a70daf9ed47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4c28652469049ca8d9d3a70daf9ed47" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39143 } }
I20260620 04:29:24.686465 22504 sys_catalog.cc:564] T 00000000000000000000000000000000 P f4c28652469049ca8d9d3a70daf9ed47 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:29:24.687458 22515 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4c28652469049ca8d9d3a70daf9ed47 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f4c28652469049ca8d9d3a70daf9ed47" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4c28652469049ca8d9d3a70daf9ed47" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39143 } } }
I20260620 04:29:24.687541 22516 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4c28652469049ca8d9d3a70daf9ed47 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f4c28652469049ca8d9d3a70daf9ed47. Latest consensus state: current_term: 1 leader_uuid: "f4c28652469049ca8d9d3a70daf9ed47" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4c28652469049ca8d9d3a70daf9ed47" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39143 } } }
I20260620 04:29:24.687580 22515 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4c28652469049ca8d9d3a70daf9ed47 [sys.catalog]: This master's current role is: LEADER
I20260620 04:29:24.687623 22516 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4c28652469049ca8d9d3a70daf9ed47 [sys.catalog]: This master's current role is: LEADER
I20260620 04:29:24.687836 22507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:24.683323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40085 (local address 127.0.98.62:39143)
0620 04:29:24.683436 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:24.683440 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:24.683645 (+ 205us) server_negotiation.cc:407] Connection header received
0620 04:29:24.683753 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:24.683756 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:24.683800 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:24.683864 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:29:24.684129 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:24.684635 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:24.685704 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:24.686036 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:24.686111 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:24.687052 (+ 941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:29:24.687055 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:29:24.687058 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:29:24.687068 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:29:24.687096 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:29:24.687110 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:29:24.687168 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 04:29:24.687481 (+ 313us) server_negotiation.cc:299] Negotiation successful
0620 04:29:24.687536 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 04:29:24.688063 22523 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:29:24.691592 22523 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:29:24.697042 22523 catalog_manager.cc:1347] Generated new cluster ID: f248b23d64c948bca1fdecbbe9e91087
I20260620 04:29:24.697103 22523 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:29:24.714447 22523 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:29:24.715365 22523 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:29:24.728761 22523 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f4c28652469049ca8d9d3a70daf9ed47: Generated new TSK 0
I20260620 04:29:24.729425 22523 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 04:29:24.735301 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 22436
2026-06-20T04:29:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1339 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781929496330398-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:29:24 dist-test-slave-nk6d krb5kdc[22535](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:29:24 dist-test-slave-nk6d krb5kdc[22535](info): set up 2 sockets
Jun 20 04:29:24 dist-test-slave-nk6d krb5kdc[22535](info): commencing operation
krb5kdc: starting...
W20260620 04:29:25.691857 392 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781929496330398-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781929496330398-392-0/krb5kdc/test-user.keytab.
Jun 20 04:29:25 dist-test-slave-nk6d krb5kdc[22535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929765, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:29:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:29:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 04:29:25.799208 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781929496330398-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781929496330398-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781929496330398-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44939
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44939
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:29:25.903421 22548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:29:25.903687 22548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:29:25.903743 22548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:29:25.907661 22548 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:29:25.907721 22548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:29:25.907747 22548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:29:25.907768 22548 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:29:25.907794 22548 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:29:25.912938 22548 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781929496330398-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781929496330398-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44939
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44939
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22548
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781929496330398-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:29:25.913901 22548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:29:25.914870 22548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:29:25.922767 22553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:29:25.922776 22556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:29:25.922775 22554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:29:25.922984 22548 server_base.cc:1034] running on GCE node
I20260620 04:29:25.923600 22548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:29:25.924654 22548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:29:25.925820 22548 hybrid_clock.cc:648] HybridClock initialized: now 1781929765925806 us; error 31 us; skew 500 ppm
Jun 20 04:29:25 dist-test-slave-nk6d krb5kdc[22535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929765, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:29:25.928442 22548 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 04:29:25.929215 22548 webserver.cc:460] Webserver started at http://127.0.98.62:36951/ using document root <none> and password file <none>
I20260620 04:29:25.929708 22548 fs_manager.cc:359] Metadata directory not provided
I20260620 04:29:25.929770 22548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:29:25.929939 22548 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:29:25.931844 22548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781929496330398-392-0/minicluster-data/master-0/data/instance:
uuid: "4e5421dd120c4d13b53d7b734f1648bf"
format_stamp: "Formatted at 2026-06-20 04:29:25 on dist-test-slave-nk6d"
server_key: "691eb3b77d9f0b7b1cc3192579513c7a"
server_key_iv: "ad77f619e7a7b025adb3ffa7da3330b3"
server_key_version: "encryptionkey@0"
I20260620 04:29:25.932401 22548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781929496330398-392-0/minicluster-data/master-0/wal/instance:
uuid: "4e5421dd120c4d13b53d7b734f1648bf"
format_stamp: "Formatted at 2026-06-20 04:29:25 on dist-test-slave-nk6d"
server_key: "691eb3b77d9f0b7b1cc3192579513c7a"
server_key_iv: "ad77f619e7a7b025adb3ffa7da3330b3"
server_key_version: "encryptionkey@0"
I20260620 04:29:25.935789 22548 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:29:25.938067 22563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:29:25.939196 22548 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:29:25.939302 22548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781929496330398-392-0/minicluster-data/master-0/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781929496330398-392-0/minicluster-data/master-0/wal
uuid: "4e5421dd120c4d13b53d7b734f1648bf"
format_stamp: "Formatted at 2026-06-20 04:29:25 on dist-test-slave-nk6d"
server_key: "691eb3b77d9f0b7b1cc3192579513c7a"
server_key_iv: "ad77f619e7a7b025adb3ffa7da3330b3"
server_key_version: "encryptionkey@0"
I20260620 04:29:25.939392 22548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781929496330398-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781929496330398-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781929496330398-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:29:25.943608 22548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:29:25.946198 22548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:29:25.946350 22548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:29:25.952551 22548 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44939
I20260620 04:29:25.952562 22615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44939 every 8 connection(s)
I20260620 04:29:25.953423 22548 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:29:25.954049 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 22548
I20260620 04:29:25.954191 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781929496330398-392-0/minicluster-data/master-0/wal/instance
I20260620 04:29:25.954465 392 external_mini_cluster.cc:1428] Setting key 4334999d57b5215136e9330f537b1650
I20260620 04:29:25.956508 22616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:29:25.959228 22619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:25.955712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40837 (local address 127.0.98.62:44939)
0620 04:29:25.956098 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:25.956113 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:25.956147 (+ 34us) server_negotiation.cc:407] Connection header received
0620 04:29:25.956788 (+ 641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:25.956805 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:25.957092 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:25.957421 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:29:25.958148 (+ 727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:29:25.958153 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:29:25.958163 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:29:25.958173 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:29:25.958224 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:29:25.958242 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:29:25.958360 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 04:29:25.958431 (+ 71us) server_negotiation.cc:299] Negotiation successful
0620 04:29:25.958574 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":94,"threads_started":1}
I20260620 04:29:25.961783 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 04:29:25.962252 22616 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e5421dd120c4d13b53d7b734f1648bf: Bootstrap starting.
I20260620 04:29:25.964504 22616 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4e5421dd120c4d13b53d7b734f1648bf: Neither blocks nor log segments found. Creating new log.
I20260620 04:29:25.965303 22619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:25.963634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59049 (local address 127.0.98.62:44939)
0620 04:29:25.963746 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:25.963750 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:25.963922 (+ 172us) server_negotiation.cc:407] Connection header received
0620 04:29:25.964030 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:25.964032 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:25.964065 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:25.964145 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:29:25.964830 (+ 685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:29:25.964833 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:29:25.964836 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:29:25.964848 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:29:25.964875 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:29:25.964887 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:29:25.964970 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 04:29:25.965146 (+ 176us) server_negotiation.cc:299] Negotiation successful
0620 04:29:25.965190 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 04:29:25.965344 22616 log.cc:824] T 00000000000000000000000000000000 P 4e5421dd120c4d13b53d7b734f1648bf: Log is configured to *not* fsync() on all Append() calls
I20260620 04:29:25.967101 22616 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e5421dd120c4d13b53d7b734f1648bf: No bootstrap required, opened a new log
I20260620 04:29:25.970420 22616 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e5421dd120c4d13b53d7b734f1648bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e5421dd120c4d13b53d7b734f1648bf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44939 } }
I20260620 04:29:25.970634 22616 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4e5421dd120c4d13b53d7b734f1648bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:29:25.970690 22616 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e5421dd120c4d13b53d7b734f1648bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e5421dd120c4d13b53d7b734f1648bf, State: Initialized, Role: FOLLOWER
I20260620 04:29:25.971117 22616 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e5421dd120c4d13b53d7b734f1648bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e5421dd120c4d13b53d7b734f1648bf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44939 } }
I20260620 04:29:25.971244 22616 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4e5421dd120c4d13b53d7b734f1648bf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:29:25.971306 22616 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4e5421dd120c4d13b53d7b734f1648bf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:29:25.971398 22616 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e5421dd120c4d13b53d7b734f1648bf [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:29:25.972287 22616 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4e5421dd120c4d13b53d7b734f1648bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e5421dd120c4d13b53d7b734f1648bf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44939 } }
I20260620 04:29:25.972594 22616 leader_election.cc:302] T 00000000000000000000000000000000 P 4e5421dd120c4d13b53d7b734f1648bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4e5421dd120c4d13b53d7b734f1648bf; no voters:
I20260620 04:29:25.972883 22616 leader_election.cc:288] T 00000000000000000000000000000000 P 4e5421dd120c4d13b53d7b734f1648bf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:29:25.972991 22626 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4e5421dd120c4d13b53d7b734f1648bf [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:29:25.973222 22626 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4e5421dd120c4d13b53d7b734f1648bf [term 1 LEADER]: Becoming Leader. State: Replica: 4e5421dd120c4d13b53d7b734f1648bf, State: Running, Role: LEADER
I20260620 04:29:25.973497 22626 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e5421dd120c4d13b53d7b734f1648bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e5421dd120c4d13b53d7b734f1648bf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44939 } }
I20260620 04:29:25.973986 22616 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e5421dd120c4d13b53d7b734f1648bf [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:29:25.975040 22627 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e5421dd120c4d13b53d7b734f1648bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4e5421dd120c4d13b53d7b734f1648bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e5421dd120c4d13b53d7b734f1648bf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44939 } } }
I20260620 04:29:25.975030 22628 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e5421dd120c4d13b53d7b734f1648bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e5421dd120c4d13b53d7b734f1648bf. Latest consensus state: current_term: 1 leader_uuid: "4e5421dd120c4d13b53d7b734f1648bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e5421dd120c4d13b53d7b734f1648bf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44939 } } }
I20260620 04:29:25.975181 22627 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e5421dd120c4d13b53d7b734f1648bf [sys.catalog]: This master's current role is: LEADER
I20260620 04:29:25.975220 22628 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e5421dd120c4d13b53d7b734f1648bf [sys.catalog]: This master's current role is: LEADER
I20260620 04:29:25.975540 22635 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:29:25.978581 22635 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:29:25.983704 22635 catalog_manager.cc:1347] Generated new cluster ID: 3e39df9521ff49d8bc5c6499acfbbca7
I20260620 04:29:25.983758 22635 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:29:26.014533 22635 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:29:26.014891 22635 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:29:26.021231 22635 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4e5421dd120c4d13b53d7b734f1648bf: Generated new TSK 0
I20260620 04:29:26.022066 22635 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 04:29:26.031271 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 22548
2026-06-20T04:29:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1295 ms)
[----------] 18 tests from AuthTokenIssuingTest (23340 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929496330398-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:29:26 dist-test-slave-nk6d krb5kdc[22647](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:29:26 dist-test-slave-nk6d krb5kdc[22647](info): set up 2 sockets
Jun 20 04:29:26 dist-test-slave-nk6d krb5kdc[22647](info): commencing operation
krb5kdc: starting...
W20260620 04:29:26.980002 392 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929496330398-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929496330398-392-0/krb5kdc/test-user.keytab.
Jun 20 04:29:27 dist-test-slave-nk6d krb5kdc[22647](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929767, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:29:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:29:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 04:29:27.091734 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929496330398-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929496330398-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929496330398-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46287
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46287,127.0.98.61:43639,127.0.98.60:39133
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:29:27.197448 22660 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 04:29:27.197714 22660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:29:27.197736 22660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:29:27.197772 22660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:29:27.201666 22660 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:29:27.201718 22660 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:29:27.201736 22660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:29:27.201750 22660 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:29:27.201763 22660 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:29:27.206826 22660 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929496330398-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929496330398-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46287,127.0.98.61:43639,127.0.98.60:39133
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46287
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22660
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929496330398-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:29:27.207779 22660 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:29:27.208806 22660 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:29:27.216732 22666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:29:27.216742 22668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:29:27.216742 22665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:29:27.217036 22660 server_base.cc:1034] running on GCE node
I20260620 04:29:27.217536 22660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:29:27.218554 22660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:29:27.219710 22660 hybrid_clock.cc:648] HybridClock initialized: now 1781929767219700 us; error 26 us; skew 500 ppm
Jun 20 04:29:27 dist-test-slave-nk6d krb5kdc[22647](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929767, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:29:27.222407 22660 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 04:29:27.223285 22660 webserver.cc:460] Webserver started at http://127.0.98.62:37085/ using document root <none> and password file <none>
I20260620 04:29:27.223789 22660 fs_manager.cc:359] Metadata directory not provided
I20260620 04:29:27.223860 22660 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:29:27.224077 22660 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:29:27.226071 22660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929496330398-392-0/minicluster-data/master-0/data/instance:
uuid: "f9920fa84a9d42c8a94aaaa6dc4ec9a1"
format_stamp: "Formatted at 2026-06-20 04:29:27 on dist-test-slave-nk6d"
server_key: "627d1951cc4fd3d014e2cbc62cac7691"
server_key_iv: "185c46c08e7a7ba845fd301b4dcb1809"
server_key_version: "encryptionkey@0"
I20260620 04:29:27.226627 22660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929496330398-392-0/minicluster-data/master-0/wal/instance:
uuid: "f9920fa84a9d42c8a94aaaa6dc4ec9a1"
format_stamp: "Formatted at 2026-06-20 04:29:27 on dist-test-slave-nk6d"
server_key: "627d1951cc4fd3d014e2cbc62cac7691"
server_key_iv: "185c46c08e7a7ba845fd301b4dcb1809"
server_key_version: "encryptionkey@0"
I20260620 04:29:27.230321 22660 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 04:29:27.232641 22675 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:29:27.233894 22660 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:29:27.234002 22660 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929496330398-392-0/minicluster-data/master-0/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929496330398-392-0/minicluster-data/master-0/wal
uuid: "f9920fa84a9d42c8a94aaaa6dc4ec9a1"
format_stamp: "Formatted at 2026-06-20 04:29:27 on dist-test-slave-nk6d"
server_key: "627d1951cc4fd3d014e2cbc62cac7691"
server_key_iv: "185c46c08e7a7ba845fd301b4dcb1809"
server_key_version: "encryptionkey@0"
I20260620 04:29:27.234097 22660 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929496330398-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929496330398-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929496330398-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:29:27.253623 22660 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:29:27.256386 22660 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:29:27.256539 22660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:29:27.263307 22660 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46287
I20260620 04:29:27.263579 22727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46287 every 8 connection(s)
I20260620 04:29:27.264273 22660 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:29:27.266973 22728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:29:27.267158 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 22660
I20260620 04:29:27.267275 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929496330398-392-0/minicluster-data/master-0/wal/instance
I20260620 04:29:27.267513 392 external_mini_cluster.cc:1428] Setting key 4857337be665f9fa3ec8e1ec06865cbb
I20260620 04:29:27.270609 22728 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46287 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Jun 20 04:29:27 dist-test-slave-nk6d krb5kdc[22647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929767, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:29:27.286347 22731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:27.273021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46287 (local address 127.0.0.1:50826)
0620 04:29:27.273495 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0620 04:29:27.273571 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0620 04:29:27.274456 (+ 885us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:29:27.275114 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:29:27.275118 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:29:27.275457 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:29:27.275726 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:29:27.275739 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:29:27.277196 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:29:27.277200 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:29:27.278274 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:29:27.278284 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:29:27.279113 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:29:27.279116 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:29:27.279192 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:27.279777 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:29:27.279794 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:29:27.281359 (+ 1565us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:29:27.283429 (+ 2070us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:29:27.283437 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:29:27.283445 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:29:27.283661 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:29:27.284068 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:29:27.284071 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:29:27.284072 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:29:27.284196 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:29:27.284881 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:29:27.284887 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:29:27.285022 (+ 135us) client_negotiation.cc:769] Sending connection context
0620 04:29:27.285205 (+ 183us) client_negotiation.cc:240] Negotiation successful
0620 04:29:27.285596 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"mutex_wait_us":113,"thread_start_us":117,"threads_started":1}
I20260620 04:29:27.286355 22732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:27.273867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50826 (local address 127.0.98.62:46287)
0620 04:29:27.274152 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:27.274163 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:27.274205 (+ 42us) server_negotiation.cc:407] Connection header received
0620 04:29:27.274754 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:27.274758 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:27.274947 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:27.275106 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:29:27.275911 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:27.277075 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:27.278403 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:27.278983 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:27.279145 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:27.281513 (+ 2368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:29:27.281520 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:29:27.281535 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:29:27.281563 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:29:27.283315 (+ 1752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:29:27.283766 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:29:27.283771 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:29:27.283773 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:29:27.283925 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:29:27.284293 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:29:27.284296 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:29:27.284298 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:29:27.284762 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:29:27.284880 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 04:29:27.285298 (+ 418us) server_negotiation.cc:299] Negotiation successful
0620 04:29:27.285484 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":184,"thread_start_us":85,"threads_started":1}
I20260620 04:29:27.288712 22728 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43639 } has no permanent_uuid. Determining permanent_uuid...
I20260620 04:29:27.289420 22731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:27.289139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43639 (local address 127.0.0.1:46636)
0620 04:29:27.289282 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 04:29:27.289365 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260620 04:29:27.289619 22678 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43639: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929496330398-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929496330398-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260620 04:29:27.290414 22731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:27.290225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43639 (local address 127.0.0.1:46652)
0620 04:29:27.290310 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0620 04:29:27.290352 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260620 04:29:27.290540 22728 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:43639: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43639: connect: Connection refused (error 111)
I20260620 04:29:27.290661 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929496330398-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929496330398-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929496330398-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929496330398-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:43639
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929496330398-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46287,127.0.98.61:43639,127.0.98.60:39133
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 04:29:27.339771 22728 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43639 } attempt: 1
I20260620 04:29:27.340649 22731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:27.340330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43639 (local address 127.0.0.1:46668)
0620 04:29:27.340485 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 04:29:27.340559 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260620 04:29:27.341336 22731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:27.341126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43639 (local address 127.0.0.1:46682)
0620 04:29:27.341218 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0620 04:29:27.341269 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260620 04:29:27.341471 22728 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:43639: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43639: connect: Connection refused (error 111)
W20260620 04:29:27.397382 22735 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 04:29:27.397655 22735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:29:27.397678 22735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:29:27.397737 22735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:29:27.401684 22735 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:29:27.401739 22735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:29:27.401757 22735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:29:27.401772 22735 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:29:27.401788 22735 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:29:27.408289 22735 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929496330398-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929496330398-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46287,127.0.98.61:43639,127.0.98.60:39133
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929496330398-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:43639
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929496330398-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22735
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929496330398-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:29:27.409634 22735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:29:27.409621 22728 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43639 } attempt: 2
I20260620 04:29:27.410463 22731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:27.410152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43639 (local address 127.0.0.1:46688)
0620 04:29:27.410303 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 04:29:27.410373 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260620 04:29:27.410948 22735 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 04:29:27.411094 22731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:27.410898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43639 (local address 127.0.0.1:46698)
0620 04:29:27.410978 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0620 04:29:27.411031 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260620 04:29:27.411206 22728 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:43639: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43639: connect: Connection refused (error 111)
W20260620 04:29:27.419179 22740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:29:27.419230 22741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:29:27.419194 22743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:29:27.419420 22735 server_base.cc:1034] running on GCE node
I20260620 04:29:27.420051 22735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:29:27.421098 22735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:29:27.422253 22735 hybrid_clock.cc:648] HybridClock initialized: now 1781929767422224 us; error 47 us; skew 500 ppm
Jun 20 04:29:27 dist-test-slave-nk6d krb5kdc[22647](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929767, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:29:27.424841 22735 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260620 04:29:27.425699 22735 webserver.cc:460] Webserver started at http://127.0.98.61:34507/ using document root <none> and password file <none>
I20260620 04:29:27.426199 22735 fs_manager.cc:359] Metadata directory not provided
I20260620 04:29:27.426254 22735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:29:27.426422 22735 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:29:27.428412 22735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929496330398-392-0/minicluster-data/master-1/data/instance:
uuid: "7ae62c71bbe44244b1f5742bee7d067b"
format_stamp: "Formatted at 2026-06-20 04:29:27 on dist-test-slave-nk6d"
server_key: "715f10b5e1e93669c8137a40ab3630e8"
server_key_iv: "70defac8904e7fc559948da57602cc13"
server_key_version: "encryptionkey@0"
I20260620 04:29:27.428906 22735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929496330398-392-0/minicluster-data/master-1/wal/instance:
uuid: "7ae62c71bbe44244b1f5742bee7d067b"
format_stamp: "Formatted at 2026-06-20 04:29:27 on dist-test-slave-nk6d"
server_key: "715f10b5e1e93669c8137a40ab3630e8"
server_key_iv: "70defac8904e7fc559948da57602cc13"
server_key_version: "encryptionkey@0"
I20260620 04:29:27.432965 22735 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 04:29:27.435215 22750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:29:27.436470 22735 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:29:27.436581 22735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929496330398-392-0/minicluster-data/master-1/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929496330398-392-0/minicluster-data/master-1/wal
uuid: "7ae62c71bbe44244b1f5742bee7d067b"
format_stamp: "Formatted at 2026-06-20 04:29:27 on dist-test-slave-nk6d"
server_key: "715f10b5e1e93669c8137a40ab3630e8"
server_key_iv: "70defac8904e7fc559948da57602cc13"
server_key_version: "encryptionkey@0"
I20260620 04:29:27.436669 22735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929496330398-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929496330398-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929496330398-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:29:27.455578 22735 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:29:27.458298 22735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:29:27.458449 22735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:29:27.465003 22735 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:43639
I20260620 04:29:27.465245 22802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:43639 every 8 connection(s)
I20260620 04:29:27.465977 22735 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929496330398-392-0/minicluster-data/master-1/data/info.pb
I20260620 04:29:27.468708 22803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:29:27.472395 22803 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46287 } has no permanent_uuid. Determining permanent_uuid...
I20260620 04:29:27.476226 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 22735
I20260620 04:29:27.476331 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929496330398-392-0/minicluster-data/master-1/wal/instance
I20260620 04:29:27.476562 392 external_mini_cluster.cc:1428] Setting key 5b753a9fcbc31c43e239506a811c1ac2
Jun 20 04:29:27 dist-test-slave-nk6d krb5kdc[22647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929767, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260620 04:29:27.485610 22732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:27.474182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50842 (local address 127.0.98.62:46287)
0620 04:29:27.474356 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:27.474361 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:27.475138 (+ 777us) server_negotiation.cc:407] Connection header received
0620 04:29:27.475982 (+ 844us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:27.475986 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:27.476044 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:27.476132 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:29:27.477321 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:27.478036 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:27.479424 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:27.479763 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:27.479874 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:27.482399 (+ 2525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:29:27.482402 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:29:27.482405 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:29:27.482435 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:29:27.483713 (+ 1278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:29:27.484264 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:29:27.484266 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:29:27.484268 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:29:27.484341 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:29:27.484629 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:29:27.484633 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:29:27.484635 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:29:27.484811 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:29:27.484888 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 04:29:27.485366 (+ 478us) server_negotiation.cc:299] Negotiation successful
0620 04:29:27.485488 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
I20260620 04:29:27.486403 22805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:27.474560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46287 (local address 127.0.0.1:50842)
0620 04:29:27.474993 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0620 04:29:27.475036 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 04:29:27.475784 (+ 748us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:29:27.476291 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:29:27.476299 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:29:27.476686 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:29:27.477173 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:29:27.477187 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:29:27.478150 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:29:27.478154 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:29:27.479303 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:29:27.479313 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:29:27.479893 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:29:27.479895 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:29:27.480077 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:27.480599 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:29:27.480615 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:29:27.482234 (+ 1619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:29:27.483838 (+ 1604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:29:27.483845 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:29:27.483860 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:29:27.484188 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:29:27.484411 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:29:27.484413 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:29:27.484415 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:29:27.484554 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:29:27.484897 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:29:27.484902 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:29:27.485130 (+ 228us) client_negotiation.cc:769] Sending connection context
0620 04:29:27.485297 (+ 167us) client_negotiation.cc:240] Negotiation successful
0620 04:29:27.485561 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":107,"threads_started":1}
I20260620 04:29:27.487767 22803 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43639 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 04:29:27 dist-test-slave-nk6d krb5kdc[22647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929767, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 04:29:27.496316 22805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:27.488247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43639 (local address 127.0.0.1:46706)
0620 04:29:27.488617 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0620 04:29:27.488630 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:29:27.488709 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:29:27.489044 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:29:27.489047 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:29:27.489206 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:29:27.489309 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:29:27.489315 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:29:27.490217 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:29:27.490221 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:29:27.491045 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:29:27.491056 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:29:27.491599 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:29:27.491603 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:29:27.491702 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:27.492171 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:29:27.492186 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:29:27.493356 (+ 1170us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:29:27.494907 (+ 1551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:29:27.494910 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:29:27.494911 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:29:27.495078 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:29:27.495358 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:29:27.495361 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:29:27.495362 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:29:27.495443 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:29:27.495835 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:29:27.495837 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:29:27.495928 (+ 91us) client_negotiation.cc:769] Sending connection context
0620 04:29:27.496052 (+ 124us) client_negotiation.cc:240] Negotiation successful
0620 04:29:27.496154 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309}
I20260620 04:29:27.496421 22808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:27.488341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46706 (local address 127.0.98.61:43639)
0620 04:29:27.488591 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:27.488599 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:27.488677 (+ 78us) server_negotiation.cc:407] Connection header received
0620 04:29:27.488781 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:27.488785 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:27.488935 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:27.489037 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:29:27.489410 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:27.490114 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:27.491144 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:27.491494 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:27.491569 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:27.493472 (+ 1903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:29:27.493480 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:29:27.493490 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:29:27.493510 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:29:27.494817 (+ 1307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:29:27.495170 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:29:27.495176 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:29:27.495178 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:29:27.495254 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:29:27.495518 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:29:27.495520 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:29:27.495521 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:29:27.495748 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:29:27.495852 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 04:29:27.496119 (+ 267us) server_negotiation.cc:299] Negotiation successful
0620 04:29:27.496239 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":92,"threads_started":1}
I20260620 04:29:27.498123 22803 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39133 } has no permanent_uuid. Determining permanent_uuid...
I20260620 04:29:27.498736 22805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:27.498514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39133 (local address 127.0.0.1:43538)
0620 04:29:27.498615 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0620 04:29:27.498681 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260620 04:29:27.498903 22752 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39133: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929496330398-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929496330398-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260620 04:29:27.499715 22805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:27.499493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39133 (local address 127.0.0.1:43540)
0620 04:29:27.499603 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0620 04:29:27.499655 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260620 04:29:27.499850 22803 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39133: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39133: connect: Connection refused (error 111)
I20260620 04:29:27.500097 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929496330398-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929496330398-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929496330398-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929496330398-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:39133
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929496330398-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46287,127.0.98.61:43639,127.0.98.60:39133
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 04:29:27.502370 22728 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43639 } attempt: 3
Jun 20 04:29:27 dist-test-slave-nk6d krb5kdc[22647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929767, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 04:29:27.510680 22731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:27.502827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43639 (local address 127.0.0.1:46710)
0620 04:29:27.502956 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 04:29:27.502971 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 04:29:27.503067 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:29:27.503363 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:29:27.503365 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:29:27.503521 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:29:27.503622 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:29:27.503627 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:29:27.504650 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:29:27.504653 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:29:27.505197 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:29:27.505205 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:29:27.505799 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:29:27.505801 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:29:27.505870 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:27.506300 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:29:27.506312 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:29:27.507467 (+ 1155us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:29:27.509331 (+ 1864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:29:27.509334 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:29:27.509335 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:29:27.509523 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:29:27.509812 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:29:27.509814 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:29:27.509815 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:29:27.509877 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:29:27.510288 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:29:27.510290 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:29:27.510367 (+ 77us) client_negotiation.cc:769] Sending connection context
0620 04:29:27.510456 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 04:29:27.510557 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260620 04:29:27.510840 22808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:27.502852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46710 (local address 127.0.98.61:43639)
0620 04:29:27.502955 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:27.502959 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:27.503027 (+ 68us) server_negotiation.cc:407] Connection header received
0620 04:29:27.503185 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:27.503188 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:27.503245 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:27.503351 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:29:27.503728 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:27.504538 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:27.505306 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:27.505684 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:27.505812 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:27.507602 (+ 1790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:29:27.507607 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:29:27.507611 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:29:27.507640 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:29:27.509224 (+ 1584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:29:27.509613 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:29:27.509617 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:29:27.509619 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:29:27.509725 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:29:27.509954 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:29:27.509958 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:29:27.509960 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:29:27.510198 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:29:27.510313 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 04:29:27.510536 (+ 223us) server_negotiation.cc:299] Negotiation successful
0620 04:29:27.510670 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 04:29:27.511824 22728 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39133 } has no permanent_uuid. Determining permanent_uuid...
I20260620 04:29:27.512554 22731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:27.512282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39133 (local address 127.0.0.1:43550)
0620 04:29:27.512395 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0620 04:29:27.512448 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 04:29:27.513427 22731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:27.513188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39133 (local address 127.0.0.1:43566)
0620 04:29:27.513326 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 04:29:27.513369 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260620 04:29:27.513531 22728 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39133: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39133: connect: Connection refused (error 111)
I20260620 04:29:27.544657 22728 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39133 } attempt: 1
I20260620 04:29:27.550467 22803 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39133 } attempt: 1
I20260620 04:29:27.551932 22805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:27.551590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39133 (local address 127.0.0.1:43580)
0620 04:29:27.551734 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 04:29:27.551813 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260620 04:29:27.552573 22731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:27.552076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39133 (local address 127.0.0.1:43590)
0620 04:29:27.552437 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0620 04:29:27.552492 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":281}
I20260620 04:29:27.552964 22805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:27.552673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39133 (local address 127.0.0.1:43602)
0620 04:29:27.552846 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 04:29:27.552900 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
W20260620 04:29:27.553108 22803 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39133: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39133: connect: Connection refused (error 111)
I20260620 04:29:27.553355 22731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:27.553063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39133 (local address 127.0.0.1:43618)
0620 04:29:27.553244 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 04:29:27.553292 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
W20260620 04:29:27.553510 22728 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39133: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39133: connect: Connection refused (error 111)
W20260620 04:29:27.615006 22810 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 04:29:27.615274 22810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:29:27.615298 22810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:29:27.615355 22810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:29:27.619287 22810 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:29:27.619338 22810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:29:27.619355 22810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:29:27.619370 22810 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:29:27.619387 22810 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:29:27.621243 22803 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39133 } attempt: 2
I20260620 04:29:27.622033 22805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:27.621740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39133 (local address 127.0.0.1:43632)
0620 04:29:27.621897 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 04:29:27.621957 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260620 04:29:27.622694 22805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:27.622500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39133 (local address 127.0.0.1:43640)
0620 04:29:27.622601 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0620 04:29:27.622642 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260620 04:29:27.622792 22803 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39133: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39133: connect: Connection refused (error 111)
I20260620 04:29:27.624854 22810 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929496330398-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929496330398-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46287,127.0.98.61:43639,127.0.98.60:39133
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929496330398-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:39133
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929496330398-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22810
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929496330398-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:29:27.625826 22810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:29:27.626796 22810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 04:29:27.628645 22728 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39133 } attempt: 2
I20260620 04:29:27.629285 22731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:27.629057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39133 (local address 127.0.0.1:43642)
0620 04:29:27.629175 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0620 04:29:27.629224 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260620 04:29:27.629909 22731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:27.629705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39133 (local address 127.0.0.1:43650)
0620 04:29:27.629806 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0620 04:29:27.629849 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260620 04:29:27.630033 22728 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39133: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39133: connect: Connection refused (error 111)
W20260620 04:29:27.634954 22818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:29:27.635016 22815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:29:27.635027 22810 server_base.cc:1034] running on GCE node
W20260620 04:29:27.635242 22816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:29:27.635737 22810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:29:27.636862 22810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:29:27.638049 22810 hybrid_clock.cc:648] HybridClock initialized: now 1781929767637994 us; error 65 us; skew 500 ppm
Jun 20 04:29:27 dist-test-slave-nk6d krb5kdc[22647](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929767, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:29:27.640825 22810 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260620 04:29:27.641716 22810 webserver.cc:460] Webserver started at http://127.0.98.60:45247/ using document root <none> and password file <none>
I20260620 04:29:27.642215 22810 fs_manager.cc:359] Metadata directory not provided
I20260620 04:29:27.642282 22810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:29:27.642481 22810 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:29:27.644868 22810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929496330398-392-0/minicluster-data/master-2/data/instance:
uuid: "d82d257702ec4b71b8642bb1b444a582"
format_stamp: "Formatted at 2026-06-20 04:29:27 on dist-test-slave-nk6d"
server_key: "0d05c1baa50668ebaf2f74a2bbe114c7"
server_key_iv: "acf69f84ee21ffb27ed2035e77e96856"
server_key_version: "encryptionkey@0"
I20260620 04:29:27.645547 22810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929496330398-392-0/minicluster-data/master-2/wal/instance:
uuid: "d82d257702ec4b71b8642bb1b444a582"
format_stamp: "Formatted at 2026-06-20 04:29:27 on dist-test-slave-nk6d"
server_key: "0d05c1baa50668ebaf2f74a2bbe114c7"
server_key_iv: "acf69f84ee21ffb27ed2035e77e96856"
server_key_version: "encryptionkey@0"
I20260620 04:29:27.649212 22810 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 04:29:27.651507 22825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:29:27.652727 22810 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 04:29:27.652837 22810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929496330398-392-0/minicluster-data/master-2/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929496330398-392-0/minicluster-data/master-2/wal
uuid: "d82d257702ec4b71b8642bb1b444a582"
format_stamp: "Formatted at 2026-06-20 04:29:27 on dist-test-slave-nk6d"
server_key: "0d05c1baa50668ebaf2f74a2bbe114c7"
server_key_iv: "acf69f84ee21ffb27ed2035e77e96856"
server_key_version: "encryptionkey@0"
I20260620 04:29:27.652930 22810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929496330398-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929496330398-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929496330398-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:29:27.664747 22810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:29:27.667276 22810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:29:27.667443 22810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:29:27.673812 22810 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:39133
I20260620 04:29:27.674063 22877 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:39133 every 8 connection(s)
I20260620 04:29:27.674697 22810 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929496330398-392-0/minicluster-data/master-2/data/info.pb
I20260620 04:29:27.676185 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 22810
I20260620 04:29:27.676291 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929496330398-392-0/minicluster-data/master-2/wal/instance
I20260620 04:29:27.676602 392 external_mini_cluster.cc:1428] Setting key 272feb908f2c42c185055e8891cb3eed
I20260620 04:29:27.677719 22878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:29:27.681728 22878 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46287 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 04:29:27 dist-test-slave-nk6d krb5kdc[22647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929767, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 20 04:29:27 dist-test-slave-nk6d krb5kdc[22647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929767, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:29:27.690088 22881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:27.677871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43658 (local address 127.0.98.60:39133)
0620 04:29:27.678241 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:27.678256 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:27.678291 (+ 35us) server_negotiation.cc:407] Connection header received
0620 04:29:27.678923 (+ 632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:27.678938 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:27.679249 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:27.679520 (+ 271us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:29:27.680409 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:27.681391 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:27.682533 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:27.682989 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:27.683107 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:27.685721 (+ 2614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:29:27.685726 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:29:27.685735 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:29:27.685763 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:29:27.687380 (+ 1617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:29:27.687778 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:29:27.687783 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:29:27.687787 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:29:27.687894 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:29:27.688159 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:29:27.688162 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:29:27.688164 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:29:27.688563 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:29:27.688770 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0620 04:29:27.689018 (+ 248us) server_negotiation.cc:299] Negotiation successful
0620 04:29:27.689312 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":103,"threads_started":1}
I20260620 04:29:27.692788 22732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:27.683526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50856 (local address 127.0.98.62:46287)
0620 04:29:27.683640 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:27.683644 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:27.684374 (+ 730us) server_negotiation.cc:407] Connection header received
0620 04:29:27.684504 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:27.684506 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:27.684550 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:27.684608 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:29:27.685247 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:27.685896 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:27.687052 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:27.687263 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:27.687387 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:27.689589 (+ 2202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:29:27.689592 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:29:27.689594 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:29:27.689620 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:29:27.690903 (+ 1283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:29:27.691339 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:29:27.691342 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:29:27.691343 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:29:27.691416 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:29:27.691878 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:29:27.691880 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:29:27.691881 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:29:27.692130 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:29:27.692227 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 04:29:27.692558 (+ 331us) server_negotiation.cc:299] Negotiation successful
0620 04:29:27.692672 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 04:29:27.692833 22882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:27.683485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46287 (local address 127.0.0.1:50856)
0620 04:29:27.684232 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0620 04:29:27.684286 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 04:29:27.684412 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:29:27.684679 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:29:27.684684 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:29:27.684992 (+ 308us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:29:27.685115 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:29:27.685120 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:29:27.686019 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:29:27.686022 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:29:27.686926 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:29:27.686937 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:29:27.687406 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:29:27.687410 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:29:27.687524 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:27.687980 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:29:27.687997 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:29:27.689455 (+ 1458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:29:27.691005 (+ 1550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:29:27.691014 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:29:27.691019 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:29:27.691221 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:29:27.691502 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:29:27.691505 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:29:27.691507 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:29:27.691590 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:29:27.692240 (+ 650us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:29:27.692246 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:29:27.692379 (+ 133us) client_negotiation.cc:769] Sending connection context
0620 04:29:27.692488 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 04:29:27.692645 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":138,"threads_started":1}
I20260620 04:29:27.693997 22878 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43639 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 04:29:27 dist-test-slave-nk6d krb5kdc[22647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929767, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Jun 20 04:29:27 dist-test-slave-nk6d krb5kdc[22647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929767, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 04:29:27.700362 22883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:27.692116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50858 (local address 127.0.98.62:46287)
0620 04:29:27.692392 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:27.692398 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:27.692407 (+ 9us) server_negotiation.cc:407] Connection header received
0620 04:29:27.692433 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:27.692435 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:27.692467 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:27.692553 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:29:27.693213 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:27.693785 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:27.694718 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:27.694939 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:27.695024 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:27.697242 (+ 2218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:29:27.697246 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:29:27.697249 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:29:27.697270 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:29:27.698671 (+ 1401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:29:27.699062 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:29:27.699065 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:29:27.699067 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:29:27.699151 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:29:27.699389 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:29:27.699392 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:29:27.699394 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:29:27.699596 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:29:27.699795 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0620 04:29:27.699905 (+ 110us) server_negotiation.cc:299] Negotiation successful
0620 04:29:27.700048 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":109,"threads_started":1}
I20260620 04:29:27.701174 22882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:27.694382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43639 (local address 127.0.0.1:46722)
0620 04:29:27.694476 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0620 04:29:27.694489 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:29:27.694553 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:29:27.694791 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:29:27.694793 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:29:27.694932 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:29:27.695014 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:29:27.695019 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:29:27.695693 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:29:27.695696 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:29:27.696248 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:29:27.696258 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:29:27.696665 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:29:27.696669 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:29:27.696772 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:27.697157 (+ 385us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:29:27.697167 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:29:27.698159 (+ 992us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:29:27.699783 (+ 1624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:29:27.699786 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:29:27.699788 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:29:27.700034 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:29:27.700299 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:29:27.700301 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:29:27.700302 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:29:27.700373 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:29:27.700751 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:29:27.700753 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:29:27.700840 (+ 87us) client_negotiation.cc:769] Sending connection context
0620 04:29:27.700921 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 04:29:27.701039 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260620 04:29:27.701304 22808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:27.694491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46722 (local address 127.0.98.61:43639)
0620 04:29:27.694609 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:27.694613 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:27.694626 (+ 13us) server_negotiation.cc:407] Connection header received
0620 04:29:27.694663 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:27.694665 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:27.694700 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:27.694760 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:29:27.695122 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:27.695598 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:27.696359 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:27.696568 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:27.696646 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:27.698293 (+ 1647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:29:27.698297 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:29:27.698299 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:29:27.698327 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:29:27.699687 (+ 1360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:29:27.700121 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:29:27.700124 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:29:27.700126 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:29:27.700207 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:29:27.700459 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:29:27.700462 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:29:27.700464 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:29:27.700665 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:29:27.700759 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 04:29:27.701070 (+ 311us) server_negotiation.cc:299] Negotiation successful
0620 04:29:27.701172 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 04:29:27.702029 22878 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39133 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 04:29:27 dist-test-slave-nk6d krb5kdc[22647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929767, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Jun 20 04:29:27 dist-test-slave-nk6d krb5kdc[22647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929767, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 04:29:27.709275 22808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:27.701524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46734 (local address 127.0.98.61:43639)
0620 04:29:27.701619 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:27.701623 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:27.701634 (+ 11us) server_negotiation.cc:407] Connection header received
0620 04:29:27.701714 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:27.701717 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:27.701757 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:27.701833 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:29:27.702435 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:27.702938 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:27.703688 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:27.703927 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:27.704042 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:27.706385 (+ 2343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:29:27.706390 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:29:27.706393 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:29:27.706420 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:29:27.707797 (+ 1377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:29:27.708199 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:29:27.708203 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:29:27.708204 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:29:27.708293 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:29:27.708556 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:29:27.708560 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:29:27.708561 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:29:27.708733 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:29:27.708962 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0620 04:29:27.709055 (+ 93us) server_negotiation.cc:299] Negotiation successful
0620 04:29:27.709139 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 04:29:27.709892 22882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:27.702446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39133 (local address 127.0.0.1:43668)
0620 04:29:27.702611 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 04:29:27.702620 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 04:29:27.702688 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:29:27.702858 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:29:27.702861 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:29:27.702999 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:29:27.703073 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:29:27.703079 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:29:27.703734 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:29:27.703737 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:29:27.704379 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:29:27.704388 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:29:27.705032 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:29:27.705036 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:29:27.705125 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:27.705481 (+ 356us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:29:27.705495 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:29:27.706645 (+ 1150us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:29:27.708289 (+ 1644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:29:27.708292 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:29:27.708294 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:29:27.708490 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:29:27.708749 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:29:27.708752 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:29:27.708754 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:29:27.708838 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:29:27.709368 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:29:27.709371 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:29:27.709473 (+ 102us) client_negotiation.cc:769] Sending connection context
0620 04:29:27.709619 (+ 146us) client_negotiation.cc:240] Negotiation successful
0620 04:29:27.709739 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 04:29:27.710294 22881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:27.702488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43668 (local address 127.0.98.60:39133)
0620 04:29:27.702570 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:27.702574 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:27.702668 (+ 94us) server_negotiation.cc:407] Connection header received
0620 04:29:27.702748 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:27.702751 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:27.702783 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:27.702840 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:29:27.703163 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:27.703636 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:27.704673 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:27.704913 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:27.705002 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:27.706769 (+ 1767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:29:27.706773 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:29:27.706776 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:29:27.706802 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:29:27.708195 (+ 1393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:29:27.708577 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:29:27.708581 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:29:27.708583 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:29:27.708660 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:29:27.708922 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:29:27.708926 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:29:27.708928 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:29:27.709189 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:29:27.709287 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 04:29:27.710066 (+ 779us) server_negotiation.cc:299] Negotiation successful
0620 04:29:27.710167 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 04:29:27.712729 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 04:29:27.713696 22878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d82d257702ec4b71b8642bb1b444a582: Bootstrap starting.
I20260620 04:29:27.713932 22803 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39133 } attempt: 3
I20260620 04:29:27.713833 22702 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "f9920fa84a9d42c8a94aaaa6dc4ec9a1"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50858
I20260620 04:29:27.716284 22878 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d82d257702ec4b71b8642bb1b444a582: Neither blocks nor log segments found. Creating new log.
I20260620 04:29:27.717109 22878 log.cc:824] T 00000000000000000000000000000000 P d82d257702ec4b71b8642bb1b444a582: Log is configured to *not* fsync() on all Append() calls
Jun 20 04:29:27 dist-test-slave-nk6d krb5kdc[22647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929767, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 04:29:27.719369 22878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d82d257702ec4b71b8642bb1b444a582: No bootstrap required, opened a new log
I20260620 04:29:27.722250 22805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:27.714267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39133 (local address 127.0.0.1:43670)
0620 04:29:27.715070 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0620 04:29:27.715084 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:29:27.715148 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:29:27.715470 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:29:27.715473 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:29:27.715626 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:29:27.715701 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:29:27.715706 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:29:27.716446 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:29:27.716450 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:29:27.716961 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:29:27.716969 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:29:27.717432 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:29:27.717435 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:29:27.717512 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:27.717977 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:29:27.717991 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:29:27.719314 (+ 1323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:29:27.720963 (+ 1649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:29:27.720966 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:29:27.720968 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:29:27.721149 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:29:27.721393 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:29:27.721396 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:29:27.721398 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:29:27.721472 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:29:27.721795 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:29:27.721798 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:29:27.721883 (+ 85us) client_negotiation.cc:769] Sending connection context
0620 04:29:27.721963 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 04:29:27.722093 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":708}
I20260620 04:29:27.722256 22881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:27.714325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43670 (local address 127.0.98.60:39133)
0620 04:29:27.715260 (+ 935us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:27.715268 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:27.715278 (+ 10us) server_negotiation.cc:407] Connection header received
0620 04:29:27.715301 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:27.715303 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:27.715351 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:27.715433 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:29:27.715800 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:27.716329 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:27.717072 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:27.717313 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:27.717432 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:27.719456 (+ 2024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:29:27.719460 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:29:27.719463 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:29:27.719504 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:29:27.720863 (+ 1359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:29:27.721232 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:29:27.721244 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:29:27.721246 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:29:27.721314 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:29:27.721547 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:29:27.721550 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:29:27.721551 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:29:27.721708 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:29:27.721791 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 04:29:27.722021 (+ 230us) server_negotiation.cc:299] Negotiation successful
0620 04:29:27.722110 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":874}
I20260620 04:29:27.723318 22878 raft_consensus.cc:348] T 00000000000000000000000000000000 P d82d257702ec4b71b8642bb1b444a582 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9920fa84a9d42c8a94aaaa6dc4ec9a1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46287 } } peers { permanent_uuid: "7ae62c71bbe44244b1f5742bee7d067b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43639 } } peers { permanent_uuid: "d82d257702ec4b71b8642bb1b444a582" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39133 } }
I20260620 04:29:27.723572 22878 raft_consensus.cc:729] T 00000000000000000000000000000000 P d82d257702ec4b71b8642bb1b444a582 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d82d257702ec4b71b8642bb1b444a582, State: Initialized, Role: FOLLOWER
I20260620 04:29:27.723901 22878 consensus_queue.cc:260] T 00000000000000000000000000000000 P d82d257702ec4b71b8642bb1b444a582 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9920fa84a9d42c8a94aaaa6dc4ec9a1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46287 } } peers { permanent_uuid: "7ae62c71bbe44244b1f5742bee7d067b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43639 } } peers { permanent_uuid: "d82d257702ec4b71b8642bb1b444a582" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39133 } }
I20260620 04:29:27.724414 22885 sys_catalog.cc:455] T 00000000000000000000000000000000 P d82d257702ec4b71b8642bb1b444a582 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9920fa84a9d42c8a94aaaa6dc4ec9a1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46287 } } peers { permanent_uuid: "7ae62c71bbe44244b1f5742bee7d067b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43639 } } peers { permanent_uuid: "d82d257702ec4b71b8642bb1b444a582" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39133 } } }
I20260620 04:29:27.724548 22885 sys_catalog.cc:458] T 00000000000000000000000000000000 P d82d257702ec4b71b8642bb1b444a582 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 04:29:27.725286 22878 sys_catalog.cc:564] T 00000000000000000000000000000000 P d82d257702ec4b71b8642bb1b444a582 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:29:27.725342 22803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ae62c71bbe44244b1f5742bee7d067b: Bootstrap starting.
I20260620 04:29:27.728221 22803 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7ae62c71bbe44244b1f5742bee7d067b: Neither blocks nor log segments found. Creating new log.
I20260620 04:29:27.729089 22803 log.cc:824] T 00000000000000000000000000000000 P 7ae62c71bbe44244b1f5742bee7d067b: Log is configured to *not* fsync() on all Append() calls
I20260620 04:29:27.729139 22728 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39133 } attempt: 3
W20260620 04:29:27.730073 22896 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d82d257702ec4b71b8642bb1b444a582: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 04:29:27.730136 22896 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 04:29:27.729079 22810 master_runner.cc:186] Error getting master registration for 127.0.98.62:46287: OK, instance_id { permanent_uuid: "f9920fa84a9d42c8a94aaaa6dc4ec9a1" instance_seqno: 1781929767261463 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 04:29:27.732865 22810 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Jun 20 04:29:27 dist-test-slave-nk6d krb5kdc[22647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929767, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 04:29:27.735451 22803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ae62c71bbe44244b1f5742bee7d067b: No bootstrap required, opened a new log
I20260620 04:29:27.738880 22731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:27.729853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39133 (local address 127.0.0.1:43686)
0620 04:29:27.730031 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 04:29:27.730046 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 04:29:27.730129 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:29:27.730444 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:29:27.730447 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:29:27.730639 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:29:27.730743 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:29:27.730750 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:29:27.731706 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:29:27.731709 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:29:27.732560 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:29:27.732573 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:29:27.733145 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:29:27.733149 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:29:27.733250 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:27.733759 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:29:27.733773 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:29:27.735133 (+ 1360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:29:27.737127 (+ 1994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:29:27.737130 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:29:27.737132 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:29:27.737399 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:29:27.737751 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:29:27.737754 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:29:27.737756 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:29:27.737845 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:29:27.738342 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:29:27.738345 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:29:27.738453 (+ 108us) client_negotiation.cc:769] Sending connection context
0620 04:29:27.738557 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 04:29:27.738714 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104}
I20260620 04:29:27.739200 22803 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7ae62c71bbe44244b1f5742bee7d067b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9920fa84a9d42c8a94aaaa6dc4ec9a1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46287 } } peers { permanent_uuid: "7ae62c71bbe44244b1f5742bee7d067b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43639 } } peers { permanent_uuid: "d82d257702ec4b71b8642bb1b444a582" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39133 } }
I20260620 04:29:27.739430 22803 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7ae62c71bbe44244b1f5742bee7d067b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ae62c71bbe44244b1f5742bee7d067b, State: Initialized, Role: FOLLOWER
I20260620 04:29:27.739450 22881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:27.729730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43686 (local address 127.0.98.60:39133)
0620 04:29:27.729829 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:27.729833 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:27.730207 (+ 374us) server_negotiation.cc:407] Connection header received
0620 04:29:27.730242 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:27.730245 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:27.730283 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:27.730344 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:29:27.730867 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:27.731568 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:27.732708 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:27.733025 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:27.733129 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:27.735277 (+ 2148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:29:27.735281 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:29:27.735283 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:29:27.735306 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:29:27.736961 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:29:27.737553 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:29:27.737556 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:29:27.737558 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:29:27.737649 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:29:27.737941 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:29:27.737945 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:29:27.737946 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:29:27.738212 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:29:27.738299 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 04:29:27.739133 (+ 834us) server_negotiation.cc:299] Negotiation successful
0620 04:29:27.739246 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 04:29:27.739768 22803 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ae62c71bbe44244b1f5742bee7d067b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9920fa84a9d42c8a94aaaa6dc4ec9a1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46287 } } peers { permanent_uuid: "7ae62c71bbe44244b1f5742bee7d067b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43639 } } peers { permanent_uuid: "d82d257702ec4b71b8642bb1b444a582" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39133 } }
I20260620 04:29:27.740440 22899 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ae62c71bbe44244b1f5742bee7d067b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9920fa84a9d42c8a94aaaa6dc4ec9a1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46287 } } peers { permanent_uuid: "7ae62c71bbe44244b1f5742bee7d067b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43639 } } peers { permanent_uuid: "d82d257702ec4b71b8642bb1b444a582" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39133 } } }
I20260620 04:29:27.740716 22899 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ae62c71bbe44244b1f5742bee7d067b [sys.catalog]: This master's current role is: FOLLOWER
I20260620 04:29:27.741294 22803 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7ae62c71bbe44244b1f5742bee7d067b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:29:27.742656 22728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9920fa84a9d42c8a94aaaa6dc4ec9a1: Bootstrap starting.
I20260620 04:29:27.745291 22728 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f9920fa84a9d42c8a94aaaa6dc4ec9a1: Neither blocks nor log segments found. Creating new log.
I20260620 04:29:27.746105 22728 log.cc:824] T 00000000000000000000000000000000 P f9920fa84a9d42c8a94aaaa6dc4ec9a1: Log is configured to *not* fsync() on all Append() calls
W20260620 04:29:27.747397 22910 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7ae62c71bbe44244b1f5742bee7d067b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 04:29:27.747502 22910 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 04:29:27.744441 22735 master_runner.cc:186] Error getting master registration for 127.0.98.62:46287: OK, instance_id { permanent_uuid: "f9920fa84a9d42c8a94aaaa6dc4ec9a1" instance_seqno: 1781929767261463 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 04:29:27.748051 22735 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 04:29:27.749766 22728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9920fa84a9d42c8a94aaaa6dc4ec9a1: No bootstrap required, opened a new log
I20260620 04:29:27.752758 22728 raft_consensus.cc:348] T 00000000000000000000000000000000 P f9920fa84a9d42c8a94aaaa6dc4ec9a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9920fa84a9d42c8a94aaaa6dc4ec9a1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46287 } } peers { permanent_uuid: "7ae62c71bbe44244b1f5742bee7d067b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43639 } } peers { permanent_uuid: "d82d257702ec4b71b8642bb1b444a582" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39133 } }
I20260620 04:29:27.752959 22728 raft_consensus.cc:729] T 00000000000000000000000000000000 P f9920fa84a9d42c8a94aaaa6dc4ec9a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9920fa84a9d42c8a94aaaa6dc4ec9a1, State: Initialized, Role: FOLLOWER
I20260620 04:29:27.753334 22728 consensus_queue.cc:260] T 00000000000000000000000000000000 P f9920fa84a9d42c8a94aaaa6dc4ec9a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9920fa84a9d42c8a94aaaa6dc4ec9a1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46287 } } peers { permanent_uuid: "7ae62c71bbe44244b1f5742bee7d067b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43639 } } peers { permanent_uuid: "d82d257702ec4b71b8642bb1b444a582" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39133 } }
I20260620 04:29:27.753918 22913 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9920fa84a9d42c8a94aaaa6dc4ec9a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9920fa84a9d42c8a94aaaa6dc4ec9a1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46287 } } peers { permanent_uuid: "7ae62c71bbe44244b1f5742bee7d067b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43639 } } peers { permanent_uuid: "d82d257702ec4b71b8642bb1b444a582" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39133 } } }
I20260620 04:29:27.754164 22913 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9920fa84a9d42c8a94aaaa6dc4ec9a1 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 04:29:27.754673 22728 sys_catalog.cc:564] T 00000000000000000000000000000000 P f9920fa84a9d42c8a94aaaa6dc4ec9a1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:29:27.754777 22702 raft_consensus.cc:482] T 00000000000000000000000000000000 P f9920fa84a9d42c8a94aaaa6dc4ec9a1 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 04:29:27.754884 22702 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f9920fa84a9d42c8a94aaaa6dc4ec9a1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:29:27.756136 22702 raft_consensus.cc:504] T 00000000000000000000000000000000 P f9920fa84a9d42c8a94aaaa6dc4ec9a1 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9920fa84a9d42c8a94aaaa6dc4ec9a1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46287 } } peers { permanent_uuid: "7ae62c71bbe44244b1f5742bee7d067b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43639 } } peers { permanent_uuid: "d82d257702ec4b71b8642bb1b444a582" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39133 } }
I20260620 04:29:27.757081 22702 leader_election.cc:288] T 00000000000000000000000000000000 P f9920fa84a9d42c8a94aaaa6dc4ec9a1 [CANDIDATE]: Term 1 election: Requested vote from peers 7ae62c71bbe44244b1f5742bee7d067b (127.0.98.61:43639), d82d257702ec4b71b8642bb1b444a582 (127.0.98.60:39133)
W20260620 04:29:27.757920 22681 tablet.cc:2369] T 00000000000000000000000000000000 P f9920fa84a9d42c8a94aaaa6dc4ec9a1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 04:29:27.758471 22852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f9920fa84a9d42c8a94aaaa6dc4ec9a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d82d257702ec4b71b8642bb1b444a582"
I20260620 04:29:27.758706 22852 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d82d257702ec4b71b8642bb1b444a582 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:29:27.760016 22852 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d82d257702ec4b71b8642bb1b444a582 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9920fa84a9d42c8a94aaaa6dc4ec9a1 in term 1.
I20260620 04:29:27.761322 22777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f9920fa84a9d42c8a94aaaa6dc4ec9a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "7ae62c71bbe44244b1f5742bee7d067b"
I20260620 04:29:27.761562 22777 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7ae62c71bbe44244b1f5742bee7d067b [term 0 FOLLOWER]: Advancing to term 1
W20260620 04:29:27.761615 22924 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f9920fa84a9d42c8a94aaaa6dc4ec9a1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 04:29:27.761675 22924 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 04:29:27.762712 22777 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7ae62c71bbe44244b1f5742bee7d067b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9920fa84a9d42c8a94aaaa6dc4ec9a1 in term 1.
I20260620 04:29:27.763218 22678 leader_election.cc:302] T 00000000000000000000000000000000 P f9920fa84a9d42c8a94aaaa6dc4ec9a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ae62c71bbe44244b1f5742bee7d067b, f9920fa84a9d42c8a94aaaa6dc4ec9a1; no voters:
I20260620 04:29:27.764992 22913 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f9920fa84a9d42c8a94aaaa6dc4ec9a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:29:27.765349 22913 raft_consensus.cc:686] T 00000000000000000000000000000000 P f9920fa84a9d42c8a94aaaa6dc4ec9a1 [term 1 LEADER]: Becoming Leader. State: Replica: f9920fa84a9d42c8a94aaaa6dc4ec9a1, State: Running, Role: LEADER
I20260620 04:29:27.765728 22913 consensus_queue.cc:237] T 00000000000000000000000000000000 P f9920fa84a9d42c8a94aaaa6dc4ec9a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9920fa84a9d42c8a94aaaa6dc4ec9a1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46287 } } peers { permanent_uuid: "7ae62c71bbe44244b1f5742bee7d067b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43639 } } peers { permanent_uuid: "d82d257702ec4b71b8642bb1b444a582" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39133 } }
I20260620 04:29:27.767900 22931 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9920fa84a9d42c8a94aaaa6dc4ec9a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f9920fa84a9d42c8a94aaaa6dc4ec9a1. Latest consensus state: current_term: 1 leader_uuid: "f9920fa84a9d42c8a94aaaa6dc4ec9a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9920fa84a9d42c8a94aaaa6dc4ec9a1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46287 } } peers { permanent_uuid: "7ae62c71bbe44244b1f5742bee7d067b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43639 } } peers { permanent_uuid: "d82d257702ec4b71b8642bb1b444a582" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39133 } } }
I20260620 04:29:27.768069 22931 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9920fa84a9d42c8a94aaaa6dc4ec9a1 [sys.catalog]: This master's current role is: LEADER
I20260620 04:29:27.769683 22660 master_runner.cc:263] No leader master found from master f9920fa84a9d42c8a94aaaa6dc4ec9a1
I20260620 04:29:27.769742 22660 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 04:29:27.774870 22808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:27.764545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46746 (local address 127.0.98.61:43639)
0620 04:29:27.764652 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:27.764656 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:27.765165 (+ 509us) server_negotiation.cc:407] Connection header received
0620 04:29:27.765341 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:27.765344 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:27.765390 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:27.765439 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:29:27.766385 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:27.767165 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:27.768302 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:27.768633 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:27.768701 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:27.771541 (+ 2840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:29:27.771545 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:29:27.771548 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:29:27.771585 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:29:27.773147 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:29:27.773632 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:29:27.773636 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:29:27.773638 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:29:27.773719 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:29:27.773990 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:29:27.773993 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:29:27.773995 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:29:27.774188 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:29:27.774281 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 04:29:27.774611 (+ 330us) server_negotiation.cc:299] Negotiation successful
0620 04:29:27.774720 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 04:29:27.780591 22933 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:29:27.782039 22933 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:29:27.788582 22883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:27.770125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50874 (local address 127.0.98.62:46287)
0620 04:29:27.770230 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:27.770235 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:27.776420 (+ 6185us) server_negotiation.cc:407] Connection header received
0620 04:29:27.776587 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:27.776591 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:27.776647 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:27.776693 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:29:27.777479 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:27.778309 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:27.779394 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:27.779766 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:27.779836 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:27.782753 (+ 2917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:29:27.782757 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:29:27.782760 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:29:27.782784 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:29:27.784605 (+ 1821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:29:27.785046 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:29:27.785050 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:29:27.785051 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:29:27.785133 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:29:27.787514 (+ 2381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:29:27.787518 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:29:27.787520 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:29:27.787743 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:29:27.787841 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 04:29:27.788292 (+ 451us) server_negotiation.cc:299] Negotiation successful
0620 04:29:27.788414 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 04:29:27.793363 22777 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7ae62c71bbe44244b1f5742bee7d067b [term 1 FOLLOWER]: Refusing update from remote peer f9920fa84a9d42c8a94aaaa6dc4ec9a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:29:27.793363 22852 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d82d257702ec4b71b8642bb1b444a582 [term 1 FOLLOWER]: Refusing update from remote peer f9920fa84a9d42c8a94aaaa6dc4ec9a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:29:27.794034 22931 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f9920fa84a9d42c8a94aaaa6dc4ec9a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ae62c71bbe44244b1f5742bee7d067b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:29:27.794471 22931 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f9920fa84a9d42c8a94aaaa6dc4ec9a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d82d257702ec4b71b8642bb1b444a582" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:29:27.797935 22885 sys_catalog.cc:455] T 00000000000000000000000000000000 P d82d257702ec4b71b8642bb1b444a582 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f9920fa84a9d42c8a94aaaa6dc4ec9a1. Latest consensus state: current_term: 1 leader_uuid: "f9920fa84a9d42c8a94aaaa6dc4ec9a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9920fa84a9d42c8a94aaaa6dc4ec9a1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46287 } } peers { permanent_uuid: "7ae62c71bbe44244b1f5742bee7d067b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43639 } } peers { permanent_uuid: "d82d257702ec4b71b8642bb1b444a582" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39133 } } }
I20260620 04:29:27.798069 22885 sys_catalog.cc:458] T 00000000000000000000000000000000 P d82d257702ec4b71b8642bb1b444a582 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 04:29:27.798349 22899 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ae62c71bbe44244b1f5742bee7d067b [sys.catalog]: SysCatalogTable state changed. Reason: New leader f9920fa84a9d42c8a94aaaa6dc4ec9a1. Latest consensus state: current_term: 1 leader_uuid: "f9920fa84a9d42c8a94aaaa6dc4ec9a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9920fa84a9d42c8a94aaaa6dc4ec9a1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46287 } } peers { permanent_uuid: "7ae62c71bbe44244b1f5742bee7d067b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43639 } } peers { permanent_uuid: "d82d257702ec4b71b8642bb1b444a582" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39133 } } }
I20260620 04:29:27.798454 22899 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ae62c71bbe44244b1f5742bee7d067b [sys.catalog]: This master's current role is: FOLLOWER
I20260620 04:29:27.802716 22931 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9920fa84a9d42c8a94aaaa6dc4ec9a1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f9920fa84a9d42c8a94aaaa6dc4ec9a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9920fa84a9d42c8a94aaaa6dc4ec9a1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46287 } } peers { permanent_uuid: "7ae62c71bbe44244b1f5742bee7d067b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43639 } } peers { permanent_uuid: "d82d257702ec4b71b8642bb1b444a582" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39133 } } }
I20260620 04:29:27.802829 22931 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9920fa84a9d42c8a94aaaa6dc4ec9a1 [sys.catalog]: This master's current role is: LEADER
I20260620 04:29:27.802970 22881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:27.770697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43698 (local address 127.0.98.60:39133)
0620 04:29:27.770803 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:27.770808 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:27.780204 (+ 9396us) server_negotiation.cc:407] Connection header received
0620 04:29:27.780394 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:27.780397 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:27.780449 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:27.780502 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:29:27.781430 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:27.782091 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:27.783136 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:27.783451 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:27.783531 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:27.799353 (+ 15822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:29:27.799357 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:29:27.799360 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:29:27.799387 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:29:27.800996 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:29:27.801446 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:29:27.801449 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:29:27.801451 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:29:27.801533 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:29:27.801832 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:29:27.801835 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:29:27.801837 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:29:27.802070 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:29:27.802170 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 04:29:27.802701 (+ 531us) server_negotiation.cc:299] Negotiation successful
0620 04:29:27.802829 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 04:29:27.804008 22936 mvcc.cc:205] Tried to move back new op lower bound from 7298784328874729472 to 7298784328772104192. Current Snapshot: MvccSnapshot[applied={T|T < 7298784328874729472}]
I20260620 04:29:27.804281 22885 sys_catalog.cc:455] T 00000000000000000000000000000000 P d82d257702ec4b71b8642bb1b444a582 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f9920fa84a9d42c8a94aaaa6dc4ec9a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9920fa84a9d42c8a94aaaa6dc4ec9a1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46287 } } peers { permanent_uuid: "7ae62c71bbe44244b1f5742bee7d067b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43639 } } peers { permanent_uuid: "d82d257702ec4b71b8642bb1b444a582" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39133 } } }
I20260620 04:29:27.804374 22885 sys_catalog.cc:458] T 00000000000000000000000000000000 P d82d257702ec4b71b8642bb1b444a582 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 04:29:27.805306 22937 mvcc.cc:205] Tried to move back new op lower bound from 7298784328874729472 to 7298784328772104192. Current Snapshot: MvccSnapshot[applied={T|T < 7298784328874729472}]
I20260620 04:29:27.806061 22913 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9920fa84a9d42c8a94aaaa6dc4ec9a1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f9920fa84a9d42c8a94aaaa6dc4ec9a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9920fa84a9d42c8a94aaaa6dc4ec9a1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46287 } } peers { permanent_uuid: "7ae62c71bbe44244b1f5742bee7d067b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43639 } } peers { permanent_uuid: "d82d257702ec4b71b8642bb1b444a582" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39133 } } }
I20260620 04:29:27.806169 22913 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9920fa84a9d42c8a94aaaa6dc4ec9a1 [sys.catalog]: This master's current role is: LEADER
I20260620 04:29:27.806989 22899 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ae62c71bbe44244b1f5742bee7d067b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f9920fa84a9d42c8a94aaaa6dc4ec9a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9920fa84a9d42c8a94aaaa6dc4ec9a1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46287 } } peers { permanent_uuid: "7ae62c71bbe44244b1f5742bee7d067b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43639 } } peers { permanent_uuid: "d82d257702ec4b71b8642bb1b444a582" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39133 } } }
I20260620 04:29:27.807106 22899 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ae62c71bbe44244b1f5742bee7d067b [sys.catalog]: This master's current role is: FOLLOWER
I20260620 04:29:27.809409 22938 mvcc.cc:205] Tried to move back new op lower bound from 7298784328874729472 to 7298784328772104192. Current Snapshot: MvccSnapshot[applied={T|T < 7298784328874729472}]
I20260620 04:29:27.813438 22933 catalog_manager.cc:1347] Generated new cluster ID: 55235ae871764fd9aaddceda59622875
I20260620 04:29:27.813503 22933 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:29:27.836529 22933 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:29:27.837910 22933 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:29:27.859731 22933 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f9920fa84a9d42c8a94aaaa6dc4ec9a1: Generated new TSK 0
I20260620 04:29:27.860860 22933 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 04:29:27.899379 22883 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:29:27.896441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50878 (local address 127.0.98.62:46287)
0620 04:29:27.896576 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:27.896581 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:27.898144 (+ 1563us) server_negotiation.cc:407] Connection header received
0620 04:29:27.898384 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:27.898388 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:27.898445 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:27.898500 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:29:27.899219 (+ 719us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50878: BlockingRecv error: recv got EOF from 127.0.0.1:50878 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260620 04:29:27.901024 22808 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:29:27.896434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46762 (local address 127.0.98.61:43639)
0620 04:29:27.896571 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:27.896576 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:27.899973 (+ 3397us) server_negotiation.cc:407] Connection header received
0620 04:29:27.900183 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:27.900186 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:27.900239 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:27.900295 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:29:27.900880 (+ 585us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46762: BlockingRecv error: recv got EOF from 127.0.0.1:46762 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260620 04:29:27.902822 22881 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:29:27.897281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43710 (local address 127.0.98.60:39133)
0620 04:29:27.897400 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:27.897404 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:27.901664 (+ 4260us) server_negotiation.cc:407] Connection header received
0620 04:29:27.902004 (+ 340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:27.902008 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:27.902071 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:27.902124 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:29:27.902702 (+ 578us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43710: BlockingRecv error: recv got EOF from 127.0.0.1:43710 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260620 04:29:27.907639 22808 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:29:27.906082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46766 (local address 127.0.98.61:43639)
0620 04:29:27.906189 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:27.906194 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:27.906676 (+ 482us) server_negotiation.cc:407] Connection header received
0620 04:29:27.906893 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:27.906897 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:27.906971 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:27.907042 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:29:27.907543 (+ 501us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46766: BlockingRecv error: recv got EOF from 127.0.0.1:46766 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260620 04:29:27.912690 22808 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:29:27.911183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46776 (local address 127.0.98.61:43639)
0620 04:29:27.911288 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:27.911291 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:27.911820 (+ 529us) server_negotiation.cc:407] Connection header received
0620 04:29:27.912038 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:27.912042 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:27.912091 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:27.912137 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:29:27.912597 (+ 460us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46776: BlockingRecv error: recv got EOF from 127.0.0.1:46776 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260620 04:29:27.918105 22808 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:29:27.916677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46790 (local address 127.0.98.61:43639)
0620 04:29:27.916770 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:27.916774 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:27.917279 (+ 505us) server_negotiation.cc:407] Connection header received
0620 04:29:27.917456 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:27.917459 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:27.917505 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:27.917552 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:29:27.918001 (+ 449us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46790: BlockingRecv error: recv got EOF from 127.0.0.1:46790 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260620 04:29:27.925689 22808 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:29:27.924108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46794 (local address 127.0.98.61:43639)
0620 04:29:27.924237 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:27.924241 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:27.924768 (+ 527us) server_negotiation.cc:407] Connection header received
0620 04:29:27.924939 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:27.924942 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:27.924989 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:27.925120 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:29:27.925587 (+ 467us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46794: BlockingRecv error: recv got EOF from 127.0.0.1:46794 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260620 04:29:27.937279 22808 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:29:27.935980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46808 (local address 127.0.98.61:43639)
0620 04:29:27.936098 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:27.936102 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:27.936501 (+ 399us) server_negotiation.cc:407] Connection header received
0620 04:29:27.936639 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:27.936643 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:27.936691 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:27.936754 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:29:27.937187 (+ 433us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46808: BlockingRecv error: recv got EOF from 127.0.0.1:46808 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260620 04:29:27.956878 22808 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:29:27.955704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46814 (local address 127.0.98.61:43639)
0620 04:29:27.955839 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:27.955844 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:27.956023 (+ 179us) server_negotiation.cc:407] Connection header received
0620 04:29:27.956176 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:27.956180 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:27.956229 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:27.956296 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:29:27.956758 (+ 462us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46814: BlockingRecv error: recv got EOF from 127.0.0.1:46814 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260620 04:29:27.992074 22808 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:29:27.990891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46816 (local address 127.0.98.61:43639)
0620 04:29:27.991046 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:27.991051 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:27.991161 (+ 110us) server_negotiation.cc:407] Connection header received
0620 04:29:27.991313 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:27.991316 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:27.991364 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:27.991431 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:29:27.991916 (+ 485us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46816: BlockingRecv error: recv got EOF from 127.0.0.1:46816 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260620 04:29:28.059448 22808 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:29:28.058252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46824 (local address 127.0.98.61:43639)
0620 04:29:28.058401 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:28.058406 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:28.058531 (+ 125us) server_negotiation.cc:407] Connection header received
0620 04:29:28.058684 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:28.058687 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:28.058735 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:28.058804 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:29:28.059346 (+ 542us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46824: BlockingRecv error: recv got EOF from 127.0.0.1:46824 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260620 04:29:28.190938 22808 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:29:28.189757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46828 (local address 127.0.98.61:43639)
0620 04:29:28.189906 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:28.189912 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:28.190093 (+ 181us) server_negotiation.cc:407] Connection header received
0620 04:29:28.190223 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:28.190226 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:28.190274 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:28.190345 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:29:28.190819 (+ 474us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46828: BlockingRecv error: recv got EOF from 127.0.0.1:46828 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260620 04:29:28.449918 22808 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:29:28.448997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46832 (local address 127.0.98.61:43639)
0620 04:29:28.449114 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:28.449118 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:28.449198 (+ 80us) server_negotiation.cc:407] Connection header received
0620 04:29:28.449315 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:28.449317 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:28.449364 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:28.449464 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:29:28.449818 (+ 354us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46832: BlockingRecv error: recv got EOF from 127.0.0.1:46832 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 04:29:28.731556 22896 catalog_manager.cc:1259] Loaded cluster ID: 55235ae871764fd9aaddceda59622875
I20260620 04:29:28.731645 22896 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d82d257702ec4b71b8642bb1b444a582: loading cluster ID for follower catalog manager: success
I20260620 04:29:28.733340 22896 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d82d257702ec4b71b8642bb1b444a582: acquiring CA information for follower catalog manager: success
I20260620 04:29:28.734460 22896 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d82d257702ec4b71b8642bb1b444a582: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 04:29:28.748752 22910 catalog_manager.cc:1259] Loaded cluster ID: 55235ae871764fd9aaddceda59622875
I20260620 04:29:28.748826 22910 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7ae62c71bbe44244b1f5742bee7d067b: loading cluster ID for follower catalog manager: success
I20260620 04:29:28.750447 22910 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7ae62c71bbe44244b1f5742bee7d067b: acquiring CA information for follower catalog manager: success
I20260620 04:29:28.751597 22910 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7ae62c71bbe44244b1f5742bee7d067b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 04:29:28.967540 23008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:28.963904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46848 (local address 127.0.98.61:43639)
0620 04:29:28.964195 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:28.964199 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:28.964214 (+ 15us) server_negotiation.cc:407] Connection header received
0620 04:29:28.964317 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:28.964320 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:28.964442 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:28.964539 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:29:28.964758 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:28.965283 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:28.966070 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:28.966294 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:28.966361 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:28.966678 (+ 317us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:29:28.967109 (+ 431us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:29:28.967224 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 04:29:28.967330 (+ 106us) server_negotiation.cc:299] Negotiation successful
0620 04:29:28.967382 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":108,"threads_started":1}
I20260620 04:29:28.973912 23014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:28.970682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43720 (local address 127.0.98.60:39133)
0620 04:29:28.970901 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:28.970904 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:28.970932 (+ 28us) server_negotiation.cc:407] Connection header received
0620 04:29:28.971004 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:28.971006 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:28.971127 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:28.971228 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:29:28.971442 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:28.971939 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:28.972738 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:28.972949 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:28.973015 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:28.973224 (+ 209us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:29:28.973628 (+ 404us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:29:28.973714 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 04:29:28.973755 (+ 41us) server_negotiation.cc:299] Negotiation successful
0620 04:29:28.973800 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":65,"threads_started":1}
I20260620 04:29:28.980675 23008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:28.976855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46860 (local address 127.0.98.61:43639)
0620 04:29:28.976961 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:28.976965 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:28.977239 (+ 274us) server_negotiation.cc:407] Connection header received
0620 04:29:28.977343 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:28.977346 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:28.977395 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:28.977461 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:29:28.977895 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:28.978656 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:28.979532 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:28.979842 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:28.979934 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:28.980197 (+ 263us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:29:28.980262 (+ 65us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:29:28.980335 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 04:29:28.980471 (+ 136us) server_negotiation.cc:299] Negotiation successful
0620 04:29:28.980528 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 04:29:28.980767 23021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:28.976647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50882 (local address 127.0.98.62:46287)
0620 04:29:28.976933 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:28.976937 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:28.976952 (+ 15us) server_negotiation.cc:407] Connection header received
0620 04:29:28.977039 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:28.977041 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:28.977204 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:28.977329 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:29:28.977550 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:28.978220 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:28.979137 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:28.979492 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:28.979600 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:28.979923 (+ 323us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:29:28.980283 (+ 360us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:29:28.980400 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 04:29:28.980469 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 04:29:28.980528 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":78,"threads_started":1}
I20260620 04:29:28.980865 23014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:28.977130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43728 (local address 127.0.98.60:39133)
0620 04:29:28.977250 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:28.977254 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:28.977523 (+ 269us) server_negotiation.cc:407] Connection header received
0620 04:29:28.977716 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:28.977720 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:28.977762 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:28.977813 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:29:28.978042 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:28.978770 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:28.979812 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:28.980130 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:28.980216 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:28.980446 (+ 230us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:29:28.980519 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:29:28.980560 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0620 04:29:28.980706 (+ 146us) server_negotiation.cc:299] Negotiation successful
0620 04:29:28.980755 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 04:29:28.988004 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 22660
I20260620 04:29:28.993637 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 22735
I20260620 04:29:28.998713 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 22810
2026-06-20T04:29:28Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2968 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929496330398-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:29:29 dist-test-slave-nk6d krb5kdc[23025](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:29:29 dist-test-slave-nk6d krb5kdc[23025](info): set up 2 sockets
Jun 20 04:29:29 dist-test-slave-nk6d krb5kdc[23025](info): commencing operation
krb5kdc: starting...
W20260620 04:29:29.952081 392 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929496330398-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929496330398-392-0/krb5kdc/test-user.keytab.
Jun 20 04:29:30 dist-test-slave-nk6d krb5kdc[23025](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929770, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:29:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:29:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 04:29:30.062366 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929496330398-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929496330398-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929496330398-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38833
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38833,127.0.98.61:42873,127.0.98.60:41719
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:29:30.171043 23038 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 04:29:30.171341 23038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:29:30.171363 23038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:29:30.171412 23038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:29:30.175419 23038 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:29:30.175477 23038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:29:30.175494 23038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:29:30.175510 23038 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:29:30.175526 23038 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:29:30.180912 23038 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929496330398-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929496330398-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38833,127.0.98.61:42873,127.0.98.60:41719
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929496330398-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38833
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23038
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929496330398-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:29:30.181955 23038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:29:30.182977 23038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:29:30.191118 23044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:29:30.191097 23046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:29:30.191267 23043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:29:30.191308 23038 server_base.cc:1034] running on GCE node
I20260620 04:29:30.192164 23038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:29:30.193266 23038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:29:30.194432 23038 hybrid_clock.cc:648] HybridClock initialized: now 1781929770194410 us; error 40 us; skew 500 ppm
Jun 20 04:29:30 dist-test-slave-nk6d krb5kdc[23025](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929770, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:29:30.197100 23038 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 04:29:30.198012 23038 webserver.cc:460] Webserver started at http://127.0.98.62:43803/ using document root <none> and password file <none>
I20260620 04:29:30.198534 23038 fs_manager.cc:359] Metadata directory not provided
I20260620 04:29:30.198602 23038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:29:30.198788 23038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:29:30.200878 23038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929496330398-392-0/minicluster-data/master-0/data/instance:
uuid: "3ac47313075b4c9aaeca38a973504a18"
format_stamp: "Formatted at 2026-06-20 04:29:30 on dist-test-slave-nk6d"
server_key: "033c3c4ac6e678677cf23d2ab53b7344"
server_key_iv: "0de602361d8f6340b38f73465034b5bb"
server_key_version: "encryptionkey@0"
I20260620 04:29:30.201423 23038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929496330398-392-0/minicluster-data/master-0/wal/instance:
uuid: "3ac47313075b4c9aaeca38a973504a18"
format_stamp: "Formatted at 2026-06-20 04:29:30 on dist-test-slave-nk6d"
server_key: "033c3c4ac6e678677cf23d2ab53b7344"
server_key_iv: "0de602361d8f6340b38f73465034b5bb"
server_key_version: "encryptionkey@0"
I20260620 04:29:30.205070 23038 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260620 04:29:30.207394 23053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:29:30.208630 23038 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:29:30.208740 23038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929496330398-392-0/minicluster-data/master-0/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929496330398-392-0/minicluster-data/master-0/wal
uuid: "3ac47313075b4c9aaeca38a973504a18"
format_stamp: "Formatted at 2026-06-20 04:29:30 on dist-test-slave-nk6d"
server_key: "033c3c4ac6e678677cf23d2ab53b7344"
server_key_iv: "0de602361d8f6340b38f73465034b5bb"
server_key_version: "encryptionkey@0"
I20260620 04:29:30.208832 23038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929496330398-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929496330398-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929496330398-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:29:30.228140 23038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:29:30.230937 23038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:29:30.231115 23038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:29:30.238125 23038 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38833
I20260620 04:29:30.238379 23105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38833 every 8 connection(s)
I20260620 04:29:30.239085 23038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929496330398-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:29:30.241909 23106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:29:30.245635 23106 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38833 } has no permanent_uuid. Determining permanent_uuid...
I20260620 04:29:30.248757 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 23038
I20260620 04:29:30.248873 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929496330398-392-0/minicluster-data/master-0/wal/instance
I20260620 04:29:30.249158 392 external_mini_cluster.cc:1428] Setting key 29161660eccc524d56d817009f11596e
Jun 20 04:29:30 dist-test-slave-nk6d krb5kdc[23025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929770, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
I20260620 04:29:30.260808 23109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:30.248439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47424 (local address 127.0.98.62:38833)
0620 04:29:30.248683 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:30.248691 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:30.248724 (+ 33us) server_negotiation.cc:407] Connection header received
0620 04:29:30.249485 (+ 761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:30.249489 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:30.249695 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:30.249840 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:29:30.250881 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:30.252186 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:30.253206 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:30.253794 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:30.253962 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:30.256122 (+ 2160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:29:30.256130 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:29:30.256144 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:29:30.256170 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:29:30.257667 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:29:30.258150 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:29:30.258155 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:29:30.258156 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:29:30.258270 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:29:30.258529 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:29:30.258531 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:29:30.258532 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:29:30.258935 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:29:30.259071 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 04:29:30.259522 (+ 451us) server_negotiation.cc:299] Negotiation successful
0620 04:29:30.259766 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":162,"thread_start_us":65,"threads_started":1}
I20260620 04:29:30.260808 23108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:30.247704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38833 (local address 127.0.0.1:47424)
0620 04:29:30.248180 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0620 04:29:30.248222 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 04:29:30.249193 (+ 971us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:29:30.249891 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:29:30.249898 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:29:30.250340 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:29:30.250724 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:29:30.250745 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:29:30.252299 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:29:30.252302 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:29:30.253092 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:29:30.253102 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:29:30.253925 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:29:30.253928 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:29:30.254022 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:30.254587 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:29:30.254602 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:29:30.255934 (+ 1332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:29:30.257781 (+ 1847us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:29:30.257789 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:29:30.257797 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:29:30.258051 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:29:30.258351 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:29:30.258353 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:29:30.258354 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:29:30.258459 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:29:30.259067 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:29:30.259073 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:29:30.259236 (+ 163us) client_negotiation.cc:769] Sending connection context
0620 04:29:30.259411 (+ 175us) client_negotiation.cc:240] Negotiation successful
0620 04:29:30.259817 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"mutex_wait_us":155,"thread_start_us":103,"threads_started":1}
I20260620 04:29:30.263325 23106 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42873 } has no permanent_uuid. Determining permanent_uuid...
I20260620 04:29:30.264108 23108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:30.263792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42873 (local address 127.0.0.1:37182)
0620 04:29:30.263915 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0620 04:29:30.264028 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 04:29:30.264302 23054 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42873: connect: Connection refused (error 111)
I20260620 04:29:30.265167 23108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:30.264936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42873 (local address 127.0.0.1:37190)
0620 04:29:30.265067 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 04:29:30.265114 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260620 04:29:30.265292 23106 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:42873: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42873: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929496330398-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929496330398-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260620 04:29:30.273108 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929496330398-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929496330398-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929496330398-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929496330398-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:42873
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929496330398-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38833,127.0.98.61:42873,127.0.98.60:41719
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 04:29:30.314515 23106 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42873 } attempt: 1
I20260620 04:29:30.315326 23108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:30.315041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42873 (local address 127.0.0.1:37192)
0620 04:29:30.315186 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 04:29:30.315250 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260620 04:29:30.316073 23108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:30.315806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42873 (local address 127.0.0.1:37206)
0620 04:29:30.315911 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0620 04:29:30.316006 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260620 04:29:30.316182 23106 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:42873: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42873: connect: Connection refused (error 111)
W20260620 04:29:30.379218 23113 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 04:29:30.379498 23113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:29:30.379532 23113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:29:30.379587 23113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:29:30.383471 23113 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:29:30.383529 23113 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:29:30.383555 23113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:29:30.383577 23113 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:29:30.383602 23113 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:29:30.384320 23106 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42873 } attempt: 2
I20260620 04:29:30.385164 23108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:30.384825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42873 (local address 127.0.0.1:37208)
0620 04:29:30.385006 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 04:29:30.385084 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260620 04:29:30.385955 23108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:30.385728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42873 (local address 127.0.0.1:37216)
0620 04:29:30.385845 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0620 04:29:30.385893 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260620 04:29:30.386096 23106 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:42873: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42873: connect: Connection refused (error 111)
I20260620 04:29:30.389117 23113 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929496330398-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929496330398-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38833,127.0.98.61:42873,127.0.98.60:41719
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929496330398-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:42873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929496330398-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23113
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929496330398-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:29:30.390126 23113 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:29:30.391119 23113 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:29:30.398960 23118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:29:30.398993 23119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:29:30.399016 23121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:29:30.399291 23113 server_base.cc:1034] running on GCE node
I20260620 04:29:30.399792 23113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:29:30.400852 23113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:29:30.402016 23113 hybrid_clock.cc:648] HybridClock initialized: now 1781929770401999 us; error 30 us; skew 500 ppm
Jun 20 04:29:30 dist-test-slave-nk6d krb5kdc[23025](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929770, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:29:30.404665 23113 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260620 04:29:30.405522 23113 webserver.cc:460] Webserver started at http://127.0.98.61:41573/ using document root <none> and password file <none>
I20260620 04:29:30.406029 23113 fs_manager.cc:359] Metadata directory not provided
I20260620 04:29:30.406092 23113 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:29:30.406267 23113 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:29:30.408308 23113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929496330398-392-0/minicluster-data/master-1/data/instance:
uuid: "5d6488dd78214a65bf526d94d1fad4a6"
format_stamp: "Formatted at 2026-06-20 04:29:30 on dist-test-slave-nk6d"
server_key: "5ea6da5da679d8c5c01694b8759a921e"
server_key_iv: "1ff349a8278df95cbe79f1e59e7dad2d"
server_key_version: "encryptionkey@0"
I20260620 04:29:30.408828 23113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929496330398-392-0/minicluster-data/master-1/wal/instance:
uuid: "5d6488dd78214a65bf526d94d1fad4a6"
format_stamp: "Formatted at 2026-06-20 04:29:30 on dist-test-slave-nk6d"
server_key: "5ea6da5da679d8c5c01694b8759a921e"
server_key_iv: "1ff349a8278df95cbe79f1e59e7dad2d"
server_key_version: "encryptionkey@0"
I20260620 04:29:30.412392 23113 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:29:30.414665 23128 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:29:30.415836 23113 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:29:30.415949 23113 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929496330398-392-0/minicluster-data/master-1/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929496330398-392-0/minicluster-data/master-1/wal
uuid: "5d6488dd78214a65bf526d94d1fad4a6"
format_stamp: "Formatted at 2026-06-20 04:29:30 on dist-test-slave-nk6d"
server_key: "5ea6da5da679d8c5c01694b8759a921e"
server_key_iv: "1ff349a8278df95cbe79f1e59e7dad2d"
server_key_version: "encryptionkey@0"
I20260620 04:29:30.416067 23113 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929496330398-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929496330398-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929496330398-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:29:30.444121 23113 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:29:30.446863 23113 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:29:30.447036 23113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:29:30.453693 23113 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:42873
I20260620 04:29:30.453940 23180 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:42873 every 8 connection(s)
I20260620 04:29:30.454646 23113 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929496330398-392-0/minicluster-data/master-1/data/info.pb
I20260620 04:29:30.457372 23181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:29:30.458657 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 23113
I20260620 04:29:30.458783 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929496330398-392-0/minicluster-data/master-1/wal/instance
I20260620 04:29:30.459100 392 external_mini_cluster.cc:1428] Setting key 748cf0778c53f2efea3cbe925fb0b834
I20260620 04:29:30.461635 23181 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38833 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Jun 20 04:29:30 dist-test-slave-nk6d krb5kdc[23025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929770, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:29:30.475224 23109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:30.463505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47430 (local address 127.0.98.62:38833)
0620 04:29:30.463625 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:30.463628 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:30.464690 (+ 1062us) server_negotiation.cc:407] Connection header received
0620 04:29:30.465684 (+ 994us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:30.465688 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:30.465751 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:30.465824 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:29:30.466974 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:30.467535 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:30.468787 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:30.469134 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:30.469242 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:30.471618 (+ 2376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:29:30.471622 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:29:30.471625 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:29:30.471654 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:29:30.473169 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:29:30.473632 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:29:30.473636 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:29:30.473638 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:29:30.473726 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:29:30.474091 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:29:30.474094 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:29:30.474096 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:29:30.474323 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:29:30.474438 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 04:29:30.474927 (+ 489us) server_negotiation.cc:299] Negotiation successful
0620 04:29:30.475053 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 04:29:30.475947 23184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:30.464042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38833 (local address 127.0.0.1:47430)
0620 04:29:30.464538 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0620 04:29:30.464582 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 04:29:30.465476 (+ 894us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:29:30.466002 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:29:30.466009 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:29:30.466337 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:29:30.466827 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:29:30.466841 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:29:30.467660 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:29:30.467662 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:29:30.468673 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:29:30.468685 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:29:30.469289 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:29:30.469292 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:29:30.469479 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:30.470083 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:29:30.470103 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:29:30.471453 (+ 1350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:29:30.473281 (+ 1828us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:29:30.473285 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:29:30.473297 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:29:30.473541 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:29:30.473850 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:29:30.473852 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:29:30.473854 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:29:30.473995 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:29:30.474415 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:29:30.474420 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:29:30.474662 (+ 242us) client_negotiation.cc:769] Sending connection context
0620 04:29:30.474843 (+ 181us) client_negotiation.cc:240] Negotiation successful
0620 04:29:30.475093 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":139,"threads_started":1}
I20260620 04:29:30.477237 23106 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42873 } attempt: 3
I20260620 04:29:30.477398 23181 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42873 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929496330398-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929496330398-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Jun 20 04:29:30 dist-test-slave-nk6d krb5kdc[23025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929770, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 04:29:30.482792 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske80qNG/build/debug/bin/kudu
/tmp/dist-test-taske80qNG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929496330398-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929496330398-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929496330398-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929496330398-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:41719
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929496330398-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38833,127.0.98.61:42873,127.0.98.60:41719
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929496330398-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929496330398-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929496330398-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 04:29:30 dist-test-slave-nk6d krb5kdc[23025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929770, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 04:29:30.486650 23184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:30.478250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42873 (local address 127.0.0.1:37238)
0620 04:29:30.478338 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0620 04:29:30.478351 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:29:30.478451 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:29:30.478763 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:29:30.478765 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:29:30.478947 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:29:30.479072 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:29:30.479077 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:29:30.479880 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:29:30.479883 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:29:30.480452 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:29:30.480461 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:29:30.481129 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:29:30.481132 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:29:30.481215 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:30.481778 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:29:30.481789 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:29:30.483457 (+ 1668us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:29:30.485214 (+ 1757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:29:30.485217 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:29:30.485219 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:29:30.485409 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:29:30.485650 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:29:30.485653 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:29:30.485655 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:29:30.485732 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:29:30.486205 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:29:30.486208 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:29:30.486287 (+ 79us) client_negotiation.cc:769] Sending connection context
0620 04:29:30.486374 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 04:29:30.486490 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27,"mutex_wait_us":31}
I20260620 04:29:30.486869 23187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:30.478310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37238 (local address 127.0.98.61:42873)
0620 04:29:30.478601 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:30.478606 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:30.478616 (+ 10us) server_negotiation.cc:407] Connection header received
0620 04:29:30.478639 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:30.478640 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:30.478669 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:30.478736 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:29:30.479169 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:30.479782 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:30.480550 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:30.481037 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:30.481155 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:30.483600 (+ 2445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:29:30.483603 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:29:30.483606 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:29:30.483626 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:29:30.485121 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:29:30.485498 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:29:30.485504 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:29:30.485505 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:29:30.485573 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:29:30.485814 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:29:30.485817 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:29:30.485819 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:29:30.486121 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:29:30.486230 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 04:29:30.486465 (+ 235us) server_negotiation.cc:299] Negotiation successful
0620 04:29:30.486582 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":238,"spinlock_wait_cycles":2560,"thread_start_us":112,"threads_started":1}
I20260620 04:29:30.487007 23108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:30.477663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42873 (local address 127.0.0.1:37226)
0620 04:29:30.477773 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0620 04:29:30.477785 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 04:29:30.477847 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:29:30.478480 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:29:30.478483 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:29:30.478649 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:29:30.478736 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:29:30.478741 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:29:30.479757 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:29:30.479761 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:29:30.480492 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:29:30.480502 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:29:30.481140 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:29:30.481144 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:29:30.481242 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:30.481698 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:29:30.481712 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:29:30.482980 (+ 1268us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:29:30.485465 (+ 2485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:29:30.485468 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:29:30.485470 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:29:30.485699 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:29:30.486050 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:29:30.486053 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:29:30.486056 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:29:30.486146 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:29:30.486558 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:29:30.486561 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:29:30.486650 (+ 89us) client_negotiation.cc:769] Sending connection context
0620 04:29:30.486729 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 04:29:30.486860 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260620 04:29:30.488319 23186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:30.477737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37226 (local address 127.0.98.61:42873)
0620 04:29:30.478079 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:30.478089 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:30.478109 (+ 20us) server_negotiation.cc:407] Connection header received
0620 04:29:30.478150 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:30.478156 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:30.478352 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:30.478480 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:29:30.478836 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:30.479647 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:30.480597 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:30.481041 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:30.481150 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:30.483123 (+ 1973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:29:30.483132 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:29:30.483146 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:29:30.483170 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:29:30.485343 (+ 2173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:29:30.485814 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:29:30.485817 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:29:30.485819 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:29:30.485951 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:29:30.486250 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:29:30.486254 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:29:30.486255 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:29:30.486456 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:29:30.487941 (+ 1485us) server_negotiation.cc:1035] Waiting for connection context
0620 04:29:30.488035 (+ 94us) server_negotiation.cc:299] Negotiation successful
0620 04:29:30.488150 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":256,"spinlock_wait_cycles":15104,"thread_start_us":135,"threads_started":1}
I20260620 04:29:30.488890 23106 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41719 } has no permanent_uuid. Determining permanent_uuid...
I20260620 04:29:30.489135 23181 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41719 } has no permanent_uuid. Determining permanent_uuid...
I20260620 04:29:30.489552 23108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:30.489314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41719 (local address 127.0.0.1:57408)
0620 04:29:30.489422 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0620 04:29:30.489473 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260620 04:29:30.489881 23184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:30.489528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41719 (local address 127.0.0.1:57422)
0620 04:29:30.489751 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0620 04:29:30.489823 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155}
W20260620 04:29:30.490079 23130 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41719: connect: Connection refused (error 111)
I20260620 04:29:30.490459 23108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:30.490181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41719 (local address 127.0.0.1:57426)
0620 04:29:30.490320 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 04:29:30.490382 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260620 04:29:30.490621 23106 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41719: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41719: connect: Connection refused (error 111)
I20260620 04:29:30.490832 23184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:30.490646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41719 (local address 127.0.0.1:57432)
0620 04:29:30.490732 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0620 04:29:30.490778 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260620 04:29:30.490955 23181 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41719: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41719: connect: Connection refused (error 111)
I20260620 04:29:30.521776 23106 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41719 } attempt: 1
I20260620 04:29:30.522614 23108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:30.522288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41719 (local address 127.0.0.1:57434)
0620 04:29:30.522449 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 04:29:30.522521 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260620 04:29:30.523280 23108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:30.523075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41719 (local address 127.0.0.1:57446)
0620 04:29:30.523167 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0620 04:29:30.523217 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260620 04:29:30.523404 23106 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41719: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41719: connect: Connection refused (error 111)
I20260620 04:29:30.540215 23181 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41719 } attempt: 1
I20260620 04:29:30.541079 23184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:30.540757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41719 (local address 127.0.0.1:57460)
0620 04:29:30.540920 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 04:29:30.540991 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260620 04:29:30.541759 23184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:30.541553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41719 (local address 127.0.0.1:57464)
0620 04:29:30.541654 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0620 04:29:30.541701 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260620 04:29:30.541898 23181 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41719: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41719: connect: Connection refused (error 111)
W20260620 04:29:30.591742 23188 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 04:29:30.592060 23188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:29:30.592103 23188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:29:30.592155 23188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:29:30.596241 23188 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:29:30.596297 23188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:29:30.596321 23188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:29:30.596344 23188 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:29:30.596372 23188 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:29:30.598546 23106 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41719 } attempt: 2
I20260620 04:29:30.599371 23108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:30.599042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41719 (local address 127.0.0.1:57466)
0620 04:29:30.599191 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 04:29:30.599258 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 04:29:30.600092 23108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:30.599861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41719 (local address 127.0.0.1:57474)
0620 04:29:30.599979 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0620 04:29:30.600027 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260620 04:29:30.600224 23106 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41719: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41719: connect: Connection refused (error 111)
I20260620 04:29:30.601984 23188 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929496330398-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929496330398-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38833,127.0.98.61:42873,127.0.98.60:41719
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929496330398-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:41719
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929496330398-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23188
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929496330398-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:29:30.602974 23188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:29:30.603945 23188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 04:29:30.610059 23181 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41719 } attempt: 2
I20260620 04:29:30.610845 23184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:30.610538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41719 (local address 127.0.0.1:57480)
0620 04:29:30.610714 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 04:29:30.610774 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 04:29:30.611769 23184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:30.611433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41719 (local address 127.0.0.1:57488)
0620 04:29:30.611612 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 04:29:30.611665 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
W20260620 04:29:30.612010 23181 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41719: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41719: connect: Connection refused (error 111)
W20260620 04:29:30.611971 23194 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:29:30.611971 23197 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:29:30.611943 23195 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:29:30.612641 23188 server_base.cc:1034] running on GCE node
I20260620 04:29:30.613160 23188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:29:30.614195 23188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:29:30.615348 23188 hybrid_clock.cc:648] HybridClock initialized: now 1781929770615321 us; error 42 us; skew 500 ppm
Jun 20 04:29:30 dist-test-slave-nk6d krb5kdc[23025](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929770, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:29:30.617898 23188 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260620 04:29:30.618758 23188 webserver.cc:460] Webserver started at http://127.0.98.60:35083/ using document root <none> and password file <none>
I20260620 04:29:30.619341 23188 fs_manager.cc:359] Metadata directory not provided
I20260620 04:29:30.619418 23188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:29:30.619606 23188 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:29:30.621609 23188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929496330398-392-0/minicluster-data/master-2/data/instance:
uuid: "c816f800f14141d7a5f48607b460c239"
format_stamp: "Formatted at 2026-06-20 04:29:30 on dist-test-slave-nk6d"
server_key: "2ab5a83ce84639b8f95b9ad175462f27"
server_key_iv: "bcfdedb96143c511b8bef2a6189a4c83"
server_key_version: "encryptionkey@0"
I20260620 04:29:30.622126 23188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929496330398-392-0/minicluster-data/master-2/wal/instance:
uuid: "c816f800f14141d7a5f48607b460c239"
format_stamp: "Formatted at 2026-06-20 04:29:30 on dist-test-slave-nk6d"
server_key: "2ab5a83ce84639b8f95b9ad175462f27"
server_key_iv: "bcfdedb96143c511b8bef2a6189a4c83"
server_key_version: "encryptionkey@0"
I20260620 04:29:30.625626 23188 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260620 04:29:30.628019 23204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:29:30.629235 23188 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:29:30.629351 23188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929496330398-392-0/minicluster-data/master-2/data,/tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929496330398-392-0/minicluster-data/master-2/wal
uuid: "c816f800f14141d7a5f48607b460c239"
format_stamp: "Formatted at 2026-06-20 04:29:30 on dist-test-slave-nk6d"
server_key: "2ab5a83ce84639b8f95b9ad175462f27"
server_key_iv: "bcfdedb96143c511b8bef2a6189a4c83"
server_key_version: "encryptionkey@0"
I20260620 04:29:30.629441 23188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929496330398-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929496330398-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929496330398-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:29:30.640236 23188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:29:30.642969 23188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:29:30.643126 23188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:29:30.649540 23188 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:41719
I20260620 04:29:30.649770 23256 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:41719 every 8 connection(s)
I20260620 04:29:30.650470 23188 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929496330398-392-0/minicluster-data/master-2/data/info.pb
I20260620 04:29:30.653196 23257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:29:30.657414 23257 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38833 } has no permanent_uuid. Determining permanent_uuid...
I20260620 04:29:30.659850 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske80qNG/build/debug/bin/kudu as pid 23188
I20260620 04:29:30.660009 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske80qNG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929496330398-392-0/minicluster-data/master-2/wal/instance
I20260620 04:29:30.660315 392 external_mini_cluster.cc:1428] Setting key 009f8216c26c1392d371b0fb5f6c050d
Jun 20 04:29:30 dist-test-slave-nk6d krb5kdc[23025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929770, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Jun 20 04:29:30 dist-test-slave-nk6d krb5kdc[23025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929770, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 04:29:30.670675 23109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:30.659120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47444 (local address 127.0.98.62:38833)
0620 04:29:30.659251 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:30.659254 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:30.660016 (+ 762us) server_negotiation.cc:407] Connection header received
0620 04:29:30.661017 (+ 1001us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:30.661021 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:30.661079 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:30.661156 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:29:30.662511 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:30.663008 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:30.664098 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:30.664399 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:30.664476 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:30.666921 (+ 2445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:29:30.666925 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:29:30.666928 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:29:30.666955 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:29:30.668413 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:29:30.668918 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:29:30.668920 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:29:30.668921 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:29:30.668992 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:29:30.669319 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:29:30.669322 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:29:30.669324 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:29:30.669545 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:29:30.669644 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 04:29:30.670222 (+ 578us) server_negotiation.cc:299] Negotiation successful
0620 04:29:30.670383 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 04:29:30.671504 23261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:30.660896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57504 (local address 127.0.98.60:41719)
0620 04:29:30.661217 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:30.661227 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:30.661281 (+ 54us) server_negotiation.cc:407] Connection header received
0620 04:29:30.661404 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:30.661408 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:30.661621 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:30.661795 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:29:30.662410 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:30.663418 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:30.664313 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:30.664741 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:30.664842 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:30.667359 (+ 2517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:29:30.667367 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:29:30.667379 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:29:30.667404 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:29:30.669012 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:29:30.669375 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:29:30.669379 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:29:30.669380 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:29:30.669448 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:29:30.669717 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:29:30.669720 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:29:30.669722 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:29:30.670013 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:29:30.670219 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0620 04:29:30.670402 (+ 183us) server_negotiation.cc:299] Negotiation successful
0620 04:29:30.670542 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":48,"server-negotiator.queue_time_us":238,"spinlock_wait_cycles":4352,"thread_start_us":132,"threads_started":1}
I20260620 04:29:30.671504 23259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:30.659425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38833 (local address 127.0.0.1:47444)
0620 04:29:30.659851 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0620 04:29:30.659899 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 04:29:30.660814 (+ 915us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:29:30.661425 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:29:30.661434 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:29:30.661888 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:29:30.662366 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:29:30.662384 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:29:30.663106 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:29:30.663110 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:29:30.663993 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:29:30.664002 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:29:30.664522 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:29:30.664526 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:29:30.664691 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:30.665283 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:29:30.665298 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:29:30.666778 (+ 1480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:29:30.668516 (+ 1738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:29:30.668523 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:29:30.668533 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:29:30.668817 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:29:30.669073 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:29:30.669076 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:29:30.669078 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:29:30.669215 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:29:30.669679 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:29:30.669687 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:29:30.669997 (+ 310us) client_negotiation.cc:769] Sending connection context
0620 04:29:30.670161 (+ 164us) client_negotiation.cc:240] Negotiation successful
0620 04:29:30.670496 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"mutex_wait_us":2,"thread_start_us":114,"threads_started":1}
I20260620 04:29:30.672806 23257 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42873 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 04:29:30 dist-test-slave-nk6d krb5kdc[23025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929770, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Jun 20 04:29:30 dist-test-slave-nk6d krb5kdc[23025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929770, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:29:30.680404 23259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:30.673246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42873 (local address 127.0.0.1:37250)
0620 04:29:30.673506 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0620 04:29:30.673520 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:29:30.673724 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:29:30.673976 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:29:30.673979 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:29:30.674120 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:29:30.674232 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:29:30.674238 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:29:30.674954 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:29:30.674957 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:29:30.675455 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:29:30.675461 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:29:30.675889 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:29:30.675892 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:29:30.676001 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:30.676413 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:29:30.676423 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:29:30.677667 (+ 1244us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:29:30.679131 (+ 1464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:29:30.679134 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:29:30.679136 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:29:30.679315 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:29:30.679551 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:29:30.679554 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:29:30.679556 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:29:30.679624 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:29:30.679977 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:29:30.679981 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:29:30.680061 (+ 80us) client_negotiation.cc:769] Sending connection context
0620 04:29:30.680143 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 04:29:30.680254 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194}
I20260620 04:29:30.680444 23186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:30.673392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37250 (local address 127.0.98.61:42873)
0620 04:29:30.673504 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:30.673507 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:30.673572 (+ 65us) server_negotiation.cc:407] Connection header received
0620 04:29:30.673845 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:30.673848 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:30.673895 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:30.673955 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:29:30.674346 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:30.674861 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:30.675562 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:30.675799 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:30.675882 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:30.677790 (+ 1908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:29:30.677794 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:29:30.677797 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:29:30.677830 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:29:30.679037 (+ 1207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:29:30.679397 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:29:30.679400 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:29:30.679402 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:29:30.679476 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:29:30.679697 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:29:30.679700 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:29:30.679702 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:29:30.679870 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:29:30.679949 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 04:29:30.680199 (+ 250us) server_negotiation.cc:299] Negotiation successful
0620 04:29:30.680315 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 04:29:30.681016 23257 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41719 } has no permanent_uuid. Determining permanent_uuid...
I20260620 04:29:30.681334 23109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:30.673660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47458 (local address 127.0.98.62:38833)
0620 04:29:30.673762 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:30.673766 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:30.673830 (+ 64us) server_negotiation.cc:407] Connection header received
0620 04:29:30.673938 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:30.673941 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:30.673980 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:30.674039 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:29:30.674562 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:30.675027 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:30.675722 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:30.675931 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:30.676032 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:30.678346 (+ 2314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:29:30.678349 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:29:30.678351 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:29:30.678371 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:29:30.679630 (+ 1259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:29:30.680039 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:29:30.680043 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:29:30.680044 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:29:30.680119 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:29:30.680352 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:29:30.680355 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:29:30.680357 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:29:30.680546 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:29:30.680760 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0620 04:29:30.681042 (+ 282us) server_negotiation.cc:299] Negotiation successful
0620 04:29:30.681187 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
Jun 20 04:29:30 dist-test-slave-nk6d krb5kdc[23025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929770, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 20 04:29:30 dist-test-slave-nk6d krb5kdc[23025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929770, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 04:29:30.688206 23259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:30.681447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41719 (local address 127.0.0.1:57518)
0620 04:29:30.681535 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0620 04:29:30.681546 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 04:29:30.681611 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:29:30.681866 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:29:30.681868 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:29:30.682019 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:29:30.682086 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:29:30.682091 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:29:30.682739 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:29:30.682742 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:29:30.683219 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:29:30.683225 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:29:30.683625 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:29:30.683628 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:29:30.683736 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:30.684174 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:29:30.684186 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:29:30.685185 (+ 999us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:29:30.686718 (+ 1533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:29:30.686722 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:29:30.686724 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:29:30.686944 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:29:30.687199 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:29:30.687202 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:29:30.687204 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:29:30.687284 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:29:30.687666 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:29:30.687669 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:29:30.687769 (+ 100us) client_negotiation.cc:769] Sending connection context
0620 04:29:30.687867 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 04:29:30.688018 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26,"mutex_wait_us":17}
I20260620 04:29:30.688213 23261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:30.681609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57518 (local address 127.0.98.60:41719)
0620 04:29:30.681697 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:30.681701 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:30.681713 (+ 12us) server_negotiation.cc:407] Connection header received
0620 04:29:30.681738 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:30.681740 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:30.681772 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:30.681836 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:29:30.682171 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:30.682644 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:30.683317 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:30.683540 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:30.683619 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:30.685303 (+ 1684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:29:30.685307 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:29:30.685309 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:29:30.685331 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:29:30.686606 (+ 1275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:29:30.687035 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:29:30.687039 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:29:30.687040 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:29:30.687124 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:29:30.687371 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:29:30.687374 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:29:30.687376 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:29:30.687572 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:29:30.687648 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 04:29:30.687930 (+ 282us) server_negotiation.cc:299] Negotiation successful
0620 04:29:30.688077 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":34}
I20260620 04:29:30.689929 23186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:30.682446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37258 (local address 127.0.98.61:42873)
0620 04:29:30.682542 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:30.682546 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:30.682556 (+ 10us) server_negotiation.cc:407] Connection header received
0620 04:29:30.682584 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:30.682586 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:30.682615 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:30.682672 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:29:30.683195 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:30.683648 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:30.684484 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:30.684717 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:30.684804 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:30.687108 (+ 2304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:29:30.687110 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:29:30.687112 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:29:30.687131 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:29:30.688532 (+ 1401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:29:30.688954 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:29:30.688957 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:29:30.688959 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:29:30.689038 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:29:30.689298 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:29:30.689300 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:29:30.689301 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:29:30.689448 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:29:30.689574 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 04:29:30.689755 (+ 181us) server_negotiation.cc:299] Negotiation successful
0620 04:29:30.689829 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 04:29:30.692075 23257 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c816f800f14141d7a5f48607b460c239: Bootstrap starting.
I20260620 04:29:30.692997 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 04:29:30.694628 23257 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c816f800f14141d7a5f48607b460c239: Neither blocks nor log segments found. Creating new log.
I20260620 04:29:30.694108 23080 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "3ac47313075b4c9aaeca38a973504a18"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47458
I20260620 04:29:30.695251 23257 log.cc:824] T 00000000000000000000000000000000 P c816f800f14141d7a5f48607b460c239: Log is configured to *not* fsync() on all Append() calls
I20260620 04:29:30.697257 23257 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c816f800f14141d7a5f48607b460c239: No bootstrap required, opened a new log
I20260620 04:29:30.699393 23106 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41719 } attempt: 3
I20260620 04:29:30.700599 23257 raft_consensus.cc:348] T 00000000000000000000000000000000 P c816f800f14141d7a5f48607b460c239 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac47313075b4c9aaeca38a973504a18" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38833 } } peers { permanent_uuid: "5d6488dd78214a65bf526d94d1fad4a6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42873 } } peers { permanent_uuid: "c816f800f14141d7a5f48607b460c239" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41719 } }
I20260620 04:29:30.700845 23257 raft_consensus.cc:729] T 00000000000000000000000000000000 P c816f800f14141d7a5f48607b460c239 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c816f800f14141d7a5f48607b460c239, State: Initialized, Role: FOLLOWER
I20260620 04:29:30.701176 23257 consensus_queue.cc:260] T 00000000000000000000000000000000 P c816f800f14141d7a5f48607b460c239 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac47313075b4c9aaeca38a973504a18" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38833 } } peers { permanent_uuid: "5d6488dd78214a65bf526d94d1fad4a6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42873 } } peers { permanent_uuid: "c816f800f14141d7a5f48607b460c239" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41719 } }
I20260620 04:29:30.701659 23263 sys_catalog.cc:455] T 00000000000000000000000000000000 P c816f800f14141d7a5f48607b460c239 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac47313075b4c9aaeca38a973504a18" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38833 } } peers { permanent_uuid: "5d6488dd78214a65bf526d94d1fad4a6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42873 } } peers { permanent_uuid: "c816f800f14141d7a5f48607b460c239" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41719 } } }
I20260620 04:29:30.701798 23263 sys_catalog.cc:458] T 00000000000000000000000000000000 P c816f800f14141d7a5f48607b460c239 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 04:29:30.702515 23257 sys_catalog.cc:564] T 00000000000000000000000000000000 P c816f800f14141d7a5f48607b460c239 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:29:30.703173 23181 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41719 } attempt: 3
Jun 20 04:29:30 dist-test-slave-nk6d krb5kdc[23025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929770, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 04:29:30.708634 23108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:30.699766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41719 (local address 127.0.0.1:57530)
0620 04:29:30.699902 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0620 04:29:30.699916 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:29:30.700029 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:29:30.700247 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:29:30.700250 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:29:30.700437 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:29:30.700541 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:29:30.700547 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:29:30.701501 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:29:30.701505 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:29:30.702178 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:29:30.702189 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:29:30.702686 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:29:30.702689 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:29:30.702776 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:30.703241 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:29:30.703255 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:29:30.704720 (+ 1465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:29:30.706639 (+ 1919us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:29:30.706642 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:29:30.706644 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:29:30.706904 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:29:30.707330 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:29:30.707333 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:29:30.707335 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:29:30.707425 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:29:30.708142 (+ 717us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:29:30.708145 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:29:30.708245 (+ 100us) client_negotiation.cc:769] Sending connection context
0620 04:29:30.708355 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 04:29:30.708484 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
W20260620 04:29:30.708735 23275 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c816f800f14141d7a5f48607b460c239: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 04:29:30.708817 23275 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Jun 20 04:29:30 dist-test-slave-nk6d krb5kdc[23025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929770, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 04:29:30.709164 23261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:30.699852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57530 (local address 127.0.98.60:41719)
0620 04:29:30.699999 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:30.700003 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:30.700016 (+ 13us) server_negotiation.cc:407] Connection header received
0620 04:29:30.700104 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:30.700107 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:30.700148 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:30.700228 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:29:30.700649 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:30.701394 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:30.702289 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:30.702584 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:30.702682 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:30.704877 (+ 2195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:29:30.704902 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:29:30.704904 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:29:30.704928 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:29:30.706524 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:29:30.707128 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:29:30.707132 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:29:30.707133 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:29:30.707218 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:29:30.707742 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:29:30.707746 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:29:30.707747 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:29:30.708008 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:29:30.708096 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 04:29:30.708926 (+ 830us) server_negotiation.cc:299] Negotiation successful
0620 04:29:30.709036 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 04:29:30.706871 23188 master_runner.cc:186] Error getting master registration for 127.0.98.62:38833: OK, instance_id { permanent_uuid: "3ac47313075b4c9aaeca38a973504a18" instance_seqno: 1781929770236167 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 04:29:30.710574 23188 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 04:29:30.712284 23106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ac47313075b4c9aaeca38a973504a18: Bootstrap starting.
I20260620 04:29:30.714932 23106 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3ac47313075b4c9aaeca38a973504a18: Neither blocks nor log segments found. Creating new log.
I20260620 04:29:30.715534 23184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:30.703549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41719 (local address 127.0.0.1:57542)
0620 04:29:30.703780 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0620 04:29:30.703794 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:29:30.703865 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:29:30.704414 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:29:30.704417 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:29:30.704590 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:29:30.704692 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:29:30.704699 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:29:30.705693 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:29:30.705696 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:29:30.706499 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:29:30.706511 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:29:30.707270 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:29:30.707274 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:29:30.707358 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:30.707767 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:29:30.707780 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:29:30.709261 (+ 1481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:29:30.713949 (+ 4688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:29:30.713952 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:29:30.713954 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:29:30.714208 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:29:30.714534 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:29:30.714537 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:29:30.714539 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:29:30.714628 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:29:30.715053 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:29:30.715056 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:29:30.715165 (+ 109us) client_negotiation.cc:769] Sending connection context
0620 04:29:30.715244 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 04:29:30.715375 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157}
I20260620 04:29:30.715765 23106 log.cc:824] T 00000000000000000000000000000000 P 3ac47313075b4c9aaeca38a973504a18: Log is configured to *not* fsync() on all Append() calls
I20260620 04:29:30.716180 23272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:30.704007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57542 (local address 127.0.98.60:41719)
0620 04:29:30.704210 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:30.704217 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:30.704230 (+ 13us) server_negotiation.cc:407] Connection header received
0620 04:29:30.704261 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:30.704263 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:30.704301 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:30.704375 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:29:30.704832 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:30.705564 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:30.706613 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:30.707114 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:30.707572 (+ 458us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:30.709463 (+ 1891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:29:30.709466 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:29:30.709491 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:29:30.709515 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:29:30.713819 (+ 4304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:29:30.714329 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:29:30.714332 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:29:30.714334 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:29:30.714433 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:29:30.714745 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:29:30.714748 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:29:30.714749 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:29:30.714944 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:29:30.715797 (+ 853us) server_negotiation.cc:1035] Waiting for connection context
0620 04:29:30.715872 (+ 75us) server_negotiation.cc:299] Negotiation successful
0620 04:29:30.716004 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":90,"threads_started":1}
I20260620 04:29:30.717815 23106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ac47313075b4c9aaeca38a973504a18: No bootstrap required, opened a new log
I20260620 04:29:30.719056 23181 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d6488dd78214a65bf526d94d1fad4a6: Bootstrap starting.
I20260620 04:29:30.720815 23106 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3ac47313075b4c9aaeca38a973504a18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac47313075b4c9aaeca38a973504a18" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38833 } } peers { permanent_uuid: "5d6488dd78214a65bf526d94d1fad4a6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42873 } } peers { permanent_uuid: "c816f800f14141d7a5f48607b460c239" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41719 } }
I20260620 04:29:30.721033 23106 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3ac47313075b4c9aaeca38a973504a18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ac47313075b4c9aaeca38a973504a18, State: Initialized, Role: FOLLOWER
I20260620 04:29:30.721362 23106 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ac47313075b4c9aaeca38a973504a18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac47313075b4c9aaeca38a973504a18" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38833 } } peers { permanent_uuid: "5d6488dd78214a65bf526d94d1fad4a6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42873 } } peers { permanent_uuid: "c816f800f14141d7a5f48607b460c239" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41719 } }
I20260620 04:29:30.721580 23181 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5d6488dd78214a65bf526d94d1fad4a6: Neither blocks nor log segments found. Creating new log.
I20260620 04:29:30.722424 23181 log.cc:824] T 00000000000000000000000000000000 P 5d6488dd78214a65bf526d94d1fad4a6: Log is configured to *not* fsync() on all Append() calls
I20260620 04:29:30.722651 23106 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ac47313075b4c9aaeca38a973504a18 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:29:30.723026 23278 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ac47313075b4c9aaeca38a973504a18 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac47313075b4c9aaeca38a973504a18" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38833 } } peers { permanent_uuid: "5d6488dd78214a65bf526d94d1fad4a6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42873 } } peers { permanent_uuid: "c816f800f14141d7a5f48607b460c239" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41719 } } }
I20260620 04:29:30.723169 23278 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ac47313075b4c9aaeca38a973504a18 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 04:29:30.724088 23080 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3ac47313075b4c9aaeca38a973504a18 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 04:29:30.724196 23080 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ac47313075b4c9aaeca38a973504a18 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:29:30.724449 23181 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d6488dd78214a65bf526d94d1fad4a6: No bootstrap required, opened a new log
I20260620 04:29:30.725174 23080 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3ac47313075b4c9aaeca38a973504a18 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac47313075b4c9aaeca38a973504a18" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38833 } } peers { permanent_uuid: "5d6488dd78214a65bf526d94d1fad4a6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42873 } } peers { permanent_uuid: "c816f800f14141d7a5f48607b460c239" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41719 } }
I20260620 04:29:30.726070 23080 leader_election.cc:288] T 00000000000000000000000000000000 P 3ac47313075b4c9aaeca38a973504a18 [CANDIDATE]: Term 1 election: Requested vote from peers 5d6488dd78214a65bf526d94d1fad4a6 (127.0.98.61:42873), c816f800f14141d7a5f48607b460c239 (127.0.98.60:41719)
I20260620 04:29:30.726701 23231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3ac47313075b4c9aaeca38a973504a18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c816f800f14141d7a5f48607b460c239"
I20260620 04:29:30.726935 23231 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c816f800f14141d7a5f48607b460c239 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:29:30.727993 23231 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c816f800f14141d7a5f48607b460c239 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ac47313075b4c9aaeca38a973504a18 in term 1.
I20260620 04:29:30.728209 23181 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5d6488dd78214a65bf526d94d1fad4a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac47313075b4c9aaeca38a973504a18" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38833 } } peers { permanent_uuid: "5d6488dd78214a65bf526d94d1fad4a6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42873 } } peers { permanent_uuid: "c816f800f14141d7a5f48607b460c239" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41719 } }
I20260620 04:29:30.728434 23181 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5d6488dd78214a65bf526d94d1fad4a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d6488dd78214a65bf526d94d1fad4a6, State: Initialized, Role: FOLLOWER
I20260620 04:29:30.728518 23055 leader_election.cc:302] T 00000000000000000000000000000000 P 3ac47313075b4c9aaeca38a973504a18 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ac47313075b4c9aaeca38a973504a18, c816f800f14141d7a5f48607b460c239; no voters:
I20260620 04:29:30.728751 23278 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3ac47313075b4c9aaeca38a973504a18 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:29:30.728739 23181 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d6488dd78214a65bf526d94d1fad4a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac47313075b4c9aaeca38a973504a18" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38833 } } peers { permanent_uuid: "5d6488dd78214a65bf526d94d1fad4a6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42873 } } peers { permanent_uuid: "c816f800f14141d7a5f48607b460c239" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41719 } }
I20260620 04:29:30.728973 23278 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3ac47313075b4c9aaeca38a973504a18 [term 1 LEADER]: Becoming Leader. State: Replica: 3ac47313075b4c9aaeca38a973504a18, State: Running, Role: LEADER
I20260620 04:29:30.729262 23278 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3ac47313075b4c9aaeca38a973504a18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac47313075b4c9aaeca38a973504a18" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38833 } } peers { permanent_uuid: "5d6488dd78214a65bf526d94d1fad4a6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42873 } } peers { permanent_uuid: "c816f800f14141d7a5f48607b460c239" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41719 } }
I20260620 04:29:30.729954 23181 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d6488dd78214a65bf526d94d1fad4a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:29:30.730841 23278 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ac47313075b4c9aaeca38a973504a18 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3ac47313075b4c9aaeca38a973504a18. Latest consensus state: current_term: 1 leader_uuid: "3ac47313075b4c9aaeca38a973504a18" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac47313075b4c9aaeca38a973504a18" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38833 } } peers { permanent_uuid: "5d6488dd78214a65bf526d94d1fad4a6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42873 } } peers { permanent_uuid: "c816f800f14141d7a5f48607b460c239" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41719 } } }
I20260620 04:29:30.730942 23278 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ac47313075b4c9aaeca38a973504a18 [sys.catalog]: This master's current role is: LEADER
I20260620 04:29:30.732048 23154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3ac47313075b4c9aaeca38a973504a18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "5d6488dd78214a65bf526d94d1fad4a6"
I20260620 04:29:30.732275 23154 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5d6488dd78214a65bf526d94d1fad4a6 [term 0 FOLLOWER]: Advancing to term 1
W20260620 04:29:30.732553 23059 tablet.cc:2369] T 00000000000000000000000000000000 P 3ac47313075b4c9aaeca38a973504a18: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 04:29:30.733532 23154 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5d6488dd78214a65bf526d94d1fad4a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ac47313075b4c9aaeca38a973504a18 in term 1.
I20260620 04:29:30.734479 23297 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d6488dd78214a65bf526d94d1fad4a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac47313075b4c9aaeca38a973504a18" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38833 } } peers { permanent_uuid: "5d6488dd78214a65bf526d94d1fad4a6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42873 } } peers { permanent_uuid: "c816f800f14141d7a5f48607b460c239" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41719 } } }
I20260620 04:29:30.734650 23297 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d6488dd78214a65bf526d94d1fad4a6 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 04:29:30.732969 23038 master_runner.cc:186] Error getting master registration for 127.0.98.61:42873: OK, instance_id { permanent_uuid: "5d6488dd78214a65bf526d94d1fad4a6" instance_seqno: 1781929770451835 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 04:29:30.736627 23038 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260620 04:29:30.737376 23290 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3ac47313075b4c9aaeca38a973504a18: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 04:29:30.737443 23290 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 04:29:30.737686 23299 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:29:30.738582 23299 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:29:30.748359 23186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:30.728873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37266 (local address 127.0.98.61:42873)
0620 04:29:30.728993 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:30.728997 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:30.734328 (+ 5331us) server_negotiation.cc:407] Connection header received
0620 04:29:30.734461 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:30.734464 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:30.734512 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:30.734607 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:29:30.740934 (+ 6327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:30.741710 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:30.742870 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:30.743077 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:30.743179 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:30.744952 (+ 1773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:29:30.744956 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:29:30.744959 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:29:30.744983 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:29:30.746568 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:29:30.747006 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:29:30.747009 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:29:30.747011 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:29:30.747096 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:29:30.747392 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:29:30.747396 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:29:30.747397 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:29:30.747615 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:29:30.747725 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 04:29:30.748075 (+ 350us) server_negotiation.cc:299] Negotiation successful
0620 04:29:30.748202 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20260620 04:29:30.751682 23310 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5d6488dd78214a65bf526d94d1fad4a6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 04:29:30.751763 23310 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 04:29:30.751771 23109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:30.743174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47468 (local address 127.0.98.62:38833)
0620 04:29:30.743281 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:30.743285 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:30.743558 (+ 273us) server_negotiation.cc:407] Connection header received
0620 04:29:30.743743 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:30.743746 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:30.743796 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:30.743844 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:29:30.744768 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:30.745457 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:30.746621 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:30.746928 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:30.746996 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:30.748619 (+ 1623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:29:30.748623 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:29:30.748625 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:29:30.748650 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:29:30.750017 (+ 1367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:29:30.750475 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:29:30.750479 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:29:30.750481 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:29:30.750561 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:29:30.750864 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:29:30.750867 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:29:30.750869 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:29:30.751068 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:29:30.751162 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 04:29:30.751488 (+ 326us) server_negotiation.cc:299] Negotiation successful
0620 04:29:30.751615 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 04:29:30.763469 23231 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c816f800f14141d7a5f48607b460c239 [term 1 FOLLOWER]: Refusing update from remote peer 3ac47313075b4c9aaeca38a973504a18: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:29:30.764180 23296 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3ac47313075b4c9aaeca38a973504a18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c816f800f14141d7a5f48607b460c239" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:29:30.764554 23272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:30.743656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57552 (local address 127.0.98.60:41719)
0620 04:29:30.743761 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:30.743765 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:30.744604 (+ 839us) server_negotiation.cc:407] Connection header received
0620 04:29:30.744768 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:30.744771 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:30.744816 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:30.744863 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:29:30.745663 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:30.746425 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:30.751892 (+ 5467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:30.752270 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:30.752342 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:30.760126 (+ 7784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:29:30.760130 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:29:30.760133 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:29:30.760164 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:29:30.762481 (+ 2317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:29:30.762967 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:29:30.762971 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:29:30.762973 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:29:30.763055 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:29:30.763338 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:29:30.763342 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:29:30.763344 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:29:30.763549 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:29:30.763674 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 04:29:30.764286 (+ 612us) server_negotiation.cc:299] Negotiation successful
0620 04:29:30.764404 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 04:29:30.768968 23263 sys_catalog.cc:455] T 00000000000000000000000000000000 P c816f800f14141d7a5f48607b460c239 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3ac47313075b4c9aaeca38a973504a18. Latest consensus state: current_term: 1 leader_uuid: "3ac47313075b4c9aaeca38a973504a18" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac47313075b4c9aaeca38a973504a18" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38833 } } peers { permanent_uuid: "5d6488dd78214a65bf526d94d1fad4a6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42873 } } peers { permanent_uuid: "c816f800f14141d7a5f48607b460c239" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41719 } } }
I20260620 04:29:30.769085 23263 sys_catalog.cc:458] T 00000000000000000000000000000000 P c816f800f14141d7a5f48607b460c239 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 04:29:30.772259 23154 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5d6488dd78214a65bf526d94d1fad4a6 [term 1 FOLLOWER]: Refusing update from remote peer 3ac47313075b4c9aaeca38a973504a18: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:29:30.772802 23296 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3ac47313075b4c9aaeca38a973504a18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d6488dd78214a65bf526d94d1fad4a6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:29:30.776525 23297 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d6488dd78214a65bf526d94d1fad4a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3ac47313075b4c9aaeca38a973504a18. Latest consensus state: current_term: 1 leader_uuid: "3ac47313075b4c9aaeca38a973504a18" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac47313075b4c9aaeca38a973504a18" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38833 } } peers { permanent_uuid: "5d6488dd78214a65bf526d94d1fad4a6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42873 } } peers { permanent_uuid: "c816f800f14141d7a5f48607b460c239" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41719 } } }
I20260620 04:29:30.776710 23297 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d6488dd78214a65bf526d94d1fad4a6 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 04:29:30.789513 23296 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ac47313075b4c9aaeca38a973504a18 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3ac47313075b4c9aaeca38a973504a18" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac47313075b4c9aaeca38a973504a18" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38833 } } peers { permanent_uuid: "5d6488dd78214a65bf526d94d1fad4a6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42873 } } peers { permanent_uuid: "c816f800f14141d7a5f48607b460c239" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41719 } } }
I20260620 04:29:30.789708 23296 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ac47313075b4c9aaeca38a973504a18 [sys.catalog]: This master's current role is: LEADER
I20260620 04:29:30.790951 23297 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d6488dd78214a65bf526d94d1fad4a6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3ac47313075b4c9aaeca38a973504a18" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac47313075b4c9aaeca38a973504a18" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38833 } } peers { permanent_uuid: "5d6488dd78214a65bf526d94d1fad4a6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42873 } } peers { permanent_uuid: "c816f800f14141d7a5f48607b460c239" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41719 } } }
I20260620 04:29:30.791172 23297 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d6488dd78214a65bf526d94d1fad4a6 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 04:29:30.791576 23312 mvcc.cc:205] Tried to move back new op lower bound from 7298784341039579136 to 7298784340910039040. Current Snapshot: MvccSnapshot[applied={T|T < 7298784341039579136}]
I20260620 04:29:30.792568 23278 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ac47313075b4c9aaeca38a973504a18 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3ac47313075b4c9aaeca38a973504a18" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac47313075b4c9aaeca38a973504a18" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38833 } } peers { permanent_uuid: "5d6488dd78214a65bf526d94d1fad4a6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42873 } } peers { permanent_uuid: "c816f800f14141d7a5f48607b460c239" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41719 } } }
I20260620 04:29:30.792694 23278 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ac47313075b4c9aaeca38a973504a18 [sys.catalog]: This master's current role is: LEADER
I20260620 04:29:30.793593 23263 sys_catalog.cc:455] T 00000000000000000000000000000000 P c816f800f14141d7a5f48607b460c239 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3ac47313075b4c9aaeca38a973504a18" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac47313075b4c9aaeca38a973504a18" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38833 } } peers { permanent_uuid: "5d6488dd78214a65bf526d94d1fad4a6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42873 } } peers { permanent_uuid: "c816f800f14141d7a5f48607b460c239" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41719 } } }
I20260620 04:29:30.793721 23263 sys_catalog.cc:458] T 00000000000000000000000000000000 P c816f800f14141d7a5f48607b460c239 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 04:29:30.797793 23316 mvcc.cc:205] Tried to move back new op lower bound from 7298784341039579136 to 7298784340910039040. Current Snapshot: MvccSnapshot[applied={T|T < 7298784341039579136}]
I20260620 04:29:30.798084 23318 mvcc.cc:205] Tried to move back new op lower bound from 7298784341039579136 to 7298784340910039040. Current Snapshot: MvccSnapshot[applied={T|T < 7298784341039579136}]
I20260620 04:29:30.799913 23299 catalog_manager.cc:1347] Generated new cluster ID: ca517c96399047d687d82be416a7f0f5
I20260620 04:29:30.800005 23299 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:29:30.826628 23299 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:29:30.828897 23299 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:29:30.844383 23299 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3ac47313075b4c9aaeca38a973504a18: Generated new TSK 0
I20260620 04:29:30.845217 23299 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 04:29:30.875263 23109 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:29:30.872226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47482 (local address 127.0.98.62:38833)
0620 04:29:30.872356 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:30.872361 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:30.874194 (+ 1833us) server_negotiation.cc:407] Connection header received
0620 04:29:30.874380 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:30.874384 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:30.874434 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:30.874487 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:29:30.875125 (+ 638us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47482: BlockingRecv error: recv got EOF from 127.0.0.1:47482 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260620 04:29:30.876683 23272 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:29:30.872986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57554 (local address 127.0.98.60:41719)
0620 04:29:30.873117 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:30.873122 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:30.875770 (+ 2648us) server_negotiation.cc:407] Connection header received
0620 04:29:30.875932 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:30.875936 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:30.876005 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:30.876056 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:29:30.876563 (+ 507us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57554: BlockingRecv error: recv got EOF from 127.0.0.1:57554 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260620 04:29:30.878021 23186 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:29:30.873598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37272 (local address 127.0.98.61:42873)
0620 04:29:30.873715 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:30.873719 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:30.877152 (+ 3433us) server_negotiation.cc:407] Connection header received
0620 04:29:30.877319 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:30.877322 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:30.877370 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:30.877419 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:29:30.877901 (+ 482us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37272: BlockingRecv error: recv got EOF from 127.0.0.1:37272 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260620 04:29:30.882479 23186 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:29:30.880913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37276 (local address 127.0.98.61:42873)
0620 04:29:30.881028 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:30.881032 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:30.881438 (+ 406us) server_negotiation.cc:407] Connection header received
0620 04:29:30.881727 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:30.881730 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:30.881810 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:30.881864 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:29:30.882377 (+ 513us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37276: BlockingRecv error: recv got EOF from 127.0.0.1:37276 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260620 04:29:30.887444 23186 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:29:30.885996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37292 (local address 127.0.98.61:42873)
0620 04:29:30.886098 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:30.886102 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:30.886588 (+ 486us) server_negotiation.cc:407] Connection header received
0620 04:29:30.886769 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:30.886772 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:30.886820 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:30.886870 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:29:30.887349 (+ 479us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37292: BlockingRecv error: recv got EOF from 127.0.0.1:37292 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260620 04:29:30.893167 23186 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:29:30.891673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37294 (local address 127.0.98.61:42873)
0620 04:29:30.891772 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:30.891776 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:30.892278 (+ 502us) server_negotiation.cc:407] Connection header received
0620 04:29:30.892465 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:30.892469 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:30.892519 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:30.892584 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:29:30.893066 (+ 482us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37294: BlockingRecv error: recv got EOF from 127.0.0.1:37294 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260620 04:29:30.900959 23186 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:29:30.899323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37306 (local address 127.0.98.61:42873)
0620 04:29:30.899453 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:30.899458 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:30.899990 (+ 532us) server_negotiation.cc:407] Connection header received
0620 04:29:30.900195 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:30.900198 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:30.900252 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:30.900304 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:29:30.900862 (+ 558us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37306: BlockingRecv error: recv got EOF from 127.0.0.1:37306 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260620 04:29:30.913137 23186 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:29:30.911338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37308 (local address 127.0.98.61:42873)
0620 04:29:30.911447 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:30.911452 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:30.912011 (+ 559us) server_negotiation.cc:407] Connection header received
0620 04:29:30.912237 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:30.912241 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:30.912293 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:30.912342 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:29:30.913022 (+ 680us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37308: BlockingRecv error: recv got EOF from 127.0.0.1:37308 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260620 04:29:30.932780 23186 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:29:30.931422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37312 (local address 127.0.98.61:42873)
0620 04:29:30.931537 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:30.931541 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:30.931865 (+ 324us) server_negotiation.cc:407] Connection header received
0620 04:29:30.932048 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:30.932051 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:30.932098 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:30.932172 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:29:30.932665 (+ 493us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37312: BlockingRecv error: recv got EOF from 127.0.0.1:37312 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
W20260620 04:29:30.967890 23186 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:29:30.966712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37326 (local address 127.0.98.61:42873)
0620 04:29:30.966856 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:30.966860 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:30.967026 (+ 166us) server_negotiation.cc:407] Connection header received
0620 04:29:30.967163 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:30.967167 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:30.967214 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:30.967281 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:29:30.967760 (+ 479us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37326: BlockingRecv error: recv got EOF from 127.0.0.1:37326 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260620 04:29:31.034963 23186 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:29:31.033888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37328 (local address 127.0.98.61:42873)
0620 04:29:31.034023 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:31.034027 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:31.034124 (+ 97us) server_negotiation.cc:407] Connection header received
0620 04:29:31.034295 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:31.034299 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:31.034351 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:31.034417 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:29:31.034847 (+ 430us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37328: BlockingRecv error: recv got EOF from 127.0.0.1:37328 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260620 04:29:31.165865 23186 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:29:31.164853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37342 (local address 127.0.98.61:42873)
0620 04:29:31.165020 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:31.165024 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:31.165113 (+ 89us) server_negotiation.cc:407] Connection header received
0620 04:29:31.165246 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:31.165249 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:31.165298 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:31.165369 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:29:31.165745 (+ 376us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37342: BlockingRecv error: recv got EOF from 127.0.0.1:37342 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260620 04:29:31.424973 23186 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:29:31.423900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37346 (local address 127.0.98.61:42873)
0620 04:29:31.424064 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:31.424068 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:31.424188 (+ 120us) server_negotiation.cc:407] Connection header received
0620 04:29:31.424308 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:31.424311 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:31.424361 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:31.424459 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:29:31.424850 (+ 391us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37346: BlockingRecv error: recv got EOF from 127.0.0.1:37346 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 04:29:31.710244 23275 catalog_manager.cc:1259] Loaded cluster ID: ca517c96399047d687d82be416a7f0f5
I20260620 04:29:31.710331 23275 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c816f800f14141d7a5f48607b460c239: loading cluster ID for follower catalog manager: success
I20260620 04:29:31.712024 23275 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c816f800f14141d7a5f48607b460c239: acquiring CA information for follower catalog manager: success
I20260620 04:29:31.713487 23275 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c816f800f14141d7a5f48607b460c239: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 04:29:31.753064 23310 catalog_manager.cc:1259] Loaded cluster ID: ca517c96399047d687d82be416a7f0f5
I20260620 04:29:31.753163 23310 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5d6488dd78214a65bf526d94d1fad4a6: loading cluster ID for follower catalog manager: success
I20260620 04:29:31.755041 23310 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5d6488dd78214a65bf526d94d1fad4a6: acquiring CA information for follower catalog manager: success
I20260620 04:29:31.756197 23310 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5d6488dd78214a65bf526d94d1fad4a6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 04:29:31.942340 23387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:31.938855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37356 (local address 127.0.98.61:42873)
0620 04:29:31.939068 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:31.939072 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:31.939130 (+ 58us) server_negotiation.cc:407] Connection header received
0620 04:29:31.939259 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:31.939262 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:31.939387 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:31.939493 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:29:31.939729 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:31.940272 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:31.941074 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:31.941297 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:31.941365 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:31.941589 (+ 224us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:29:31.942012 (+ 423us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:29:31.942099 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 04:29:31.942123 (+ 24us) server_negotiation.cc:299] Negotiation successful
0620 04:29:31.942170 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":73,"threads_started":1}
I20260620 04:29:31.948175 23393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:31.945110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57560 (local address 127.0.98.60:41719)
0620 04:29:31.945326 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:31.945329 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:31.945344 (+ 15us) server_negotiation.cc:407] Connection header received
0620 04:29:31.945416 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:31.945418 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:31.945524 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:31.945614 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:29:31.945759 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:31.946243 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:31.946962 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:31.947164 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:31.947229 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:31.947431 (+ 202us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:29:31.947836 (+ 405us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:29:31.947922 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 04:29:31.947971 (+ 49us) server_negotiation.cc:299] Negotiation successful
0620 04:29:31.948027 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":77,"threads_started":1}
I20260620 04:29:31.954609 23393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:31.951255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57574 (local address 127.0.98.60:41719)
0620 04:29:31.951366 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:31.951371 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:31.951659 (+ 288us) server_negotiation.cc:407] Connection header received
0620 04:29:31.951812 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:31.951815 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:31.951855 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:31.951911 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:29:31.952185 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:31.952887 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:31.953681 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:31.953930 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:31.954021 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:31.954213 (+ 192us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:29:31.954289 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:29:31.954350 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 04:29:31.954440 (+ 90us) server_negotiation.cc:299] Negotiation successful
0620 04:29:31.954488 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 04:29:31.955145 23387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:31.950919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37364 (local address 127.0.98.61:42873)
0620 04:29:31.951161 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:31.951165 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:31.951449 (+ 284us) server_negotiation.cc:407] Connection header received
0620 04:29:31.951517 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:31.951520 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:31.951565 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:31.951645 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:29:31.952213 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:31.952688 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:31.953899 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:31.954193 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:31.954307 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:31.954513 (+ 206us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:29:31.954589 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:29:31.954663 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 04:29:31.954984 (+ 321us) server_negotiation.cc:299] Negotiation successful
0620 04:29:31.955035 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162}
I20260620 04:29:31.955816 23400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:29:31.950941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47484 (local address 127.0.98.62:38833)
0620 04:29:31.951302 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0620 04:29:31.951307 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:29:31.951322 (+ 15us) server_negotiation.cc:407] Connection header received
0620 04:29:31.951411 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:29:31.951413 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:29:31.951557 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:29:31.951765 (+ 208us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:29:31.951937 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:31.952716 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:31.954376 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:29:31.954683 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:29:31.955029 (+ 346us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:29:31.955130 (+ 101us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:29:31.955420 (+ 290us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:29:31.955537 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 04:29:31.955630 (+ 93us) server_negotiation.cc:299] Negotiation successful
0620 04:29:31.955684 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":70,"threads_started":1}
I20260620 04:29:31.958366 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 23038
I20260620 04:29:31.963465 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 23113
I20260620 04:29:31.968148 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske80qNG/build/debug/bin/kudu with pid 23188
2026-06-20T04:29:31Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2969 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5937 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (275629 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST